Log file created at: 2026/11/21 10:32:26 Running on machine: ls Binary: Built with gc go1.23.8 for linux/amd64 Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg I1121 10:32:26.441946 3798194 net.go:642] Add success. I1121 10:32:26.449491 3798194 net.go:778] primary dev: ETH0 I1121 10:32:26.449507 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:32:26.449521 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:32:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:32:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:32:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:32:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:32:36.412257 3798194 memory.go:184] no items to output this cycle I1121 10:32:36.412410 3798194 cpu.go:275] no items to output this cycle E1121 10:32:46.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:32:46.411986 3798194 memory.go:184] no items to output this cycle I1121 10:32:46.412156 3798194 cpu.go:275] no items to output this cycle E1121 10:32:56.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:32:56.412885 3798194 memory.go:191] Add success. W1121 10:32:56.412917 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:32:56.412929 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:32:56.412932 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:32:56.417369 3798194 cpu.go:282] Add success. I1121 10:32:56.441805 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:32:56.442352 3798194 net.go:642] Add success. I1121 10:32:56.443918 3798194 net.go:778] primary dev: ETH0 I1121 10:32:56.443933 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:32:56.443945 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:32:56.467710 3798194 disk_info.go:125] begin check local disk info of client I1121 10:32:56.471362 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:32:56.471370 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e9a40 0xc0002e9a80] I1121 10:32:57.443681 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:32:57.445304 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:32:57.445318 3798194 disk_worker.go:723] disk space is not compliant W1121 10:32:57.445321 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:32:57.447787 3798194 disk_worker.go:509] system disk:vda1 I1121 10:32:57.447830 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:32:58.446371 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:32:59.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:32:59.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:32:59.450151 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:32:59.471678 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:33:06.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:33:06.412889 3798194 memory.go:184] no items to output this cycle I1121 10:33:06.413036 3798194 cpu.go:275] no items to output this cycle E1121 10:33:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:33:16.412098 3798194 memory.go:184] no items to output this cycle I1121 10:33:16.412463 3798194 cpu.go:275] no items to output this cycle I1121 10:33:17.438708 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 10:33:17.438717 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 10:33:26.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:33:26.413261 3798194 memory.go:191] Add success. I1121 10:33:26.417772 3798194 cpu.go:282] Add success. I1121 10:33:26.441284 3798194 net.go:642] Add success. I1121 10:33:26.442941 3798194 net.go:778] primary dev: ETH0 I1121 10:33:26.442963 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:33:26.442984 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:33:29.450410 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:33:29.450488 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:33:29.450519 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:33:36.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:33:36.412259 3798194 memory.go:184] no items to output this cycle I1121 10:33:36.412400 3798194 cpu.go:275] no items to output this cycle E1121 10:33:46.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:33:46.413267 3798194 memory.go:184] no items to output this cycle I1121 10:33:46.413410 3798194 cpu.go:275] no items to output this cycle E1121 10:33:56.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:33:56.412263 3798194 memory.go:191] Add success. W1121 10:33:56.412296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:33:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:33:56.412311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:33:56.416549 3798194 cpu.go:282] Add success. I1121 10:33:56.441953 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:33:56.442536 3798194 net.go:642] Add success. I1121 10:33:56.444144 3798194 net.go:778] primary dev: ETH0 I1121 10:33:56.444167 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:33:56.444183 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:33:56.472107 3798194 disk_info.go:125] begin check local disk info of client I1121 10:33:56.475668 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:33:56.475676 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000306780 0xc0003067c0] I1121 10:33:57.455862 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:33:57.456166 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:33:57.456183 3798194 disk_worker.go:723] disk space is not compliant W1121 10:33:57.456187 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:33:57.468436 3798194 disk_worker.go:509] system disk:vda1 I1121 10:33:57.468504 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:33:58.446547 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:33:59.450652 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:33:59.450733 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:33:59.450759 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:33:59.471267 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:34:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:34:06.412259 3798194 memory.go:184] no items to output this cycle I1121 10:34:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 10:34:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:34:16.411981 3798194 memory.go:184] no items to output this cycle I1121 10:34:16.412137 3798194 cpu.go:275] no items to output this cycle E1121 10:34:26.412872 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:34:26.412907 3798194 memory.go:191] Add success. I1121 10:34:26.417528 3798194 cpu.go:282] Add success. I1121 10:34:26.442086 3798194 net.go:642] Add success. I1121 10:34:26.449182 3798194 net.go:778] primary dev: ETH0 I1121 10:34:26.449197 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:34:26.449311 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:34:29.449895 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:34:29.449972 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:34:29.450001 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:34:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:34:36.411980 3798194 memory.go:184] no items to output this cycle I1121 10:34:36.412134 3798194 cpu.go:275] no items to output this cycle E1121 10:34:46.413251 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:34:46.413273 3798194 memory.go:184] no items to output this cycle I1121 10:34:46.413428 3798194 cpu.go:275] no items to output this cycle E1121 10:34:56.412868 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:34:56.412914 3798194 memory.go:191] Add success. W1121 10:34:56.412943 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:34:56.412955 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:34:56.412958 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:34:56.417174 3798194 cpu.go:282] Add success. I1121 10:34:56.441660 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:34:56.442191 3798194 net.go:642] Add success. I1121 10:34:56.443834 3798194 net.go:778] primary dev: ETH0 I1121 10:34:56.443849 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:34:56.443861 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:34:56.476730 3798194 disk_info.go:125] begin check local disk info of client I1121 10:34:56.480642 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:34:56.480650 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000367d00 0xc000367d40] I1121 10:34:57.450927 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:34:57.451161 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:34:57.451173 3798194 disk_worker.go:723] disk space is not compliant W1121 10:34:57.451177 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:34:57.459926 3798194 disk_worker.go:509] system disk:vda1 I1121 10:34:57.459967 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:34:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:34:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:34:59.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:34:59.450502 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:34:59.471025 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:35:06.412265 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:35:06.412287 3798194 memory.go:184] no items to output this cycle I1121 10:35:06.412430 3798194 cpu.go:275] no items to output this cycle E1121 10:35:16.412868 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:35:16.412895 3798194 memory.go:184] no items to output this cycle I1121 10:35:16.413042 3798194 cpu.go:275] no items to output this cycle E1121 10:35:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:35:26.411952 3798194 memory.go:191] Add success. I1121 10:35:26.412296 3798194 cpu.go:282] Add success. I1121 10:35:26.441756 3798194 net.go:642] Add success. I1121 10:35:26.446323 3798194 net.go:778] primary dev: ETH0 I1121 10:35:26.446342 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:35:26.446356 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:35:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:35:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:35:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:35:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:35:36.413256 3798194 memory.go:184] no items to output this cycle I1121 10:35:36.413400 3798194 cpu.go:275] no items to output this cycle E1121 10:35:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:35:46.411979 3798194 memory.go:184] no items to output this cycle I1121 10:35:46.412146 3798194 cpu.go:275] no items to output this cycle E1121 10:35:56.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:35:56.413260 3798194 memory.go:191] Add success. W1121 10:35:56.413286 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:35:56.413298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:35:56.413302 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:35:56.418169 3798194 cpu.go:282] Add success. I1121 10:35:56.442598 3798194 event_worker.go:152] Polling the log file for events... I1121 10:35:56.442939 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:35:56.443519 3798194 net.go:642] Add success. I1121 10:35:56.445555 3798194 net.go:778] primary dev: ETH0 I1121 10:35:56.445570 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:35:56.445582 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:35:56.445601 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795226756, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 10:35:56.453054 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 I1121 10:35:56.481277 3798194 disk_info.go:125] begin check local disk info of client I1121 10:35:56.485186 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:35:56.485193 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000487a40 0xc000487a80] E1121 10:35:57.445586 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 10:35:57.445603 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:35:57.445794 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:35:57.445852 3798194 disk_worker.go:723] disk space is not compliant W1121 10:35:57.445855 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:35:57.447947 3798194 disk_worker.go:509] system disk:vda1 I1121 10:35:57.447986 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 10:35:58.448589 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 10:35:58.448604 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 10:35:59.448988 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 10:35:59.450064 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:35:59.450126 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:35:59.450146 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:35:59.471671 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:36:06.412870 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:36:06.412903 3798194 memory.go:184] no items to output this cycle I1121 10:36:06.413045 3798194 cpu.go:275] no items to output this cycle E1121 10:36:16.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:36:16.412875 3798194 memory.go:184] no items to output this cycle I1121 10:36:16.413036 3798194 cpu.go:275] no items to output this cycle I1121 10:36:17.439034 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 10:36:17.439042 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 10:36:26.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:36:26.413284 3798194 memory.go:191] Add success. I1121 10:36:26.417832 3798194 cpu.go:282] Add success. I1121 10:36:26.441288 3798194 net.go:642] Add success. I1121 10:36:26.442827 3798194 net.go:778] primary dev: ETH0 I1121 10:36:26.442842 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:36:26.442854 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:36:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:36:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:36:29.450147 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:36:36.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:36:36.411985 3798194 memory.go:184] no items to output this cycle I1121 10:36:36.412146 3798194 cpu.go:275] no items to output this cycle E1121 10:36:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:36:46.411979 3798194 memory.go:184] no items to output this cycle I1121 10:36:46.412135 3798194 cpu.go:275] no items to output this cycle E1121 10:36:56.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:36:56.413264 3798194 memory.go:191] Add success. W1121 10:36:56.413290 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:36:56.413302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:36:56.413306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:36:56.417859 3798194 cpu.go:282] Add success. I1121 10:36:56.441277 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:36:56.441762 3798194 net.go:642] Add success. I1121 10:36:56.443228 3798194 net.go:778] primary dev: ETH0 I1121 10:36:56.443243 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:36:56.443257 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:36:56.486127 3798194 disk_info.go:125] begin check local disk info of client I1121 10:36:56.489978 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:36:56.489990 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000440740 0xc000440780] I1121 10:36:57.450337 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:36:57.450543 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:36:57.450554 3798194 disk_worker.go:723] disk space is not compliant W1121 10:36:57.450558 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:36:57.459785 3798194 disk_worker.go:509] system disk:vda1 I1121 10:36:57.459826 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:36:58.446367 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:36:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:36:59.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:36:59.450510 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:36:59.471075 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:37:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:37:06.411976 3798194 memory.go:184] no items to output this cycle I1121 10:37:06.412132 3798194 cpu.go:275] no items to output this cycle E1121 10:37:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:37:16.411976 3798194 memory.go:184] no items to output this cycle I1121 10:37:16.412140 3798194 cpu.go:275] no items to output this cycle E1121 10:37:26.411197 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:37:26.411233 3798194 memory.go:191] Add success. I1121 10:37:26.415777 3798194 cpu.go:282] Add success. I1121 10:37:26.441298 3798194 net.go:642] Add success. I1121 10:37:26.442808 3798194 net.go:778] primary dev: ETH0 I1121 10:37:26.442822 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:37:26.442835 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:37:29.450489 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:37:29.450568 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:37:29.450596 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:37:36.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:37:36.412262 3798194 memory.go:184] no items to output this cycle I1121 10:37:36.412405 3798194 cpu.go:275] no items to output this cycle E1121 10:37:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:37:46.413260 3798194 memory.go:184] no items to output this cycle I1121 10:37:46.413413 3798194 cpu.go:275] no items to output this cycle E1121 10:37:56.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:37:56.412267 3798194 memory.go:191] Add success. W1121 10:37:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:37:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:37:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:37:56.416744 3798194 cpu.go:282] Add success. I1121 10:37:56.441151 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:37:56.441707 3798194 net.go:642] Add success. I1121 10:37:56.443301 3798194 net.go:778] primary dev: ETH0 I1121 10:37:56.443316 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:37:56.443328 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:37:56.490095 3798194 disk_info.go:125] begin check local disk info of client I1121 10:37:56.499830 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:37:56.499839 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036b280 0xc00036b2c0] I1121 10:37:57.459013 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:37:57.459248 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:37:57.459259 3798194 disk_worker.go:723] disk space is not compliant W1121 10:37:57.459262 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:37:57.472425 3798194 disk_worker.go:509] system disk:vda1 I1121 10:37:57.472468 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:37:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:37:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:37:59.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:37:59.450499 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:37:59.471000 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:38:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:38:06.412253 3798194 memory.go:184] no items to output this cycle I1121 10:38:06.412395 3798194 cpu.go:275] no items to output this cycle E1121 10:38:16.412072 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:38:16.412096 3798194 memory.go:184] no items to output this cycle I1121 10:38:16.412402 3798194 cpu.go:275] no items to output this cycle E1121 10:38:26.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:38:26.411946 3798194 memory.go:191] Add success. I1121 10:38:26.412311 3798194 cpu.go:282] Add success. I1121 10:38:26.441857 3798194 net.go:642] Add success. I1121 10:38:26.445826 3798194 net.go:778] primary dev: ETH0 I1121 10:38:26.445841 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:38:26.445855 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:38:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:38:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:38:29.450129 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:38:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:38:36.412253 3798194 memory.go:184] no items to output this cycle I1121 10:38:36.412397 3798194 cpu.go:275] no items to output this cycle E1121 10:38:46.413272 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:38:46.413297 3798194 memory.go:184] no items to output this cycle I1121 10:38:46.413452 3798194 cpu.go:275] no items to output this cycle E1121 10:38:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:38:56.411954 3798194 memory.go:191] Add success. W1121 10:38:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:38:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:38:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:38:56.412330 3798194 cpu.go:282] Add success. I1121 10:38:56.441800 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:38:56.442398 3798194 net.go:642] Add success. I1121 10:38:56.446424 3798194 net.go:778] primary dev: ETH0 I1121 10:38:56.446440 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:38:56.446453 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:38:56.500438 3798194 disk_info.go:125] begin check local disk info of client I1121 10:38:56.504006 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:38:56.504017 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002fe740 0xc0002fe780] I1121 10:38:57.450996 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:38:57.451190 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:38:57.451243 3798194 disk_worker.go:723] disk space is not compliant W1121 10:38:57.451246 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:38:57.460002 3798194 disk_worker.go:509] system disk:vda1 I1121 10:38:57.460042 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:38:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:38:59.450108 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:38:59.450193 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:38:59.450217 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:38:59.470728 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:39:06.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:39:06.412267 3798194 memory.go:184] no items to output this cycle I1121 10:39:06.412410 3798194 cpu.go:275] no items to output this cycle E1121 10:39:16.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:39:16.412256 3798194 memory.go:184] no items to output this cycle I1121 10:39:16.412405 3798194 cpu.go:275] no items to output this cycle I1121 10:39:17.439410 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 10:39:17.439418 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 10:39:26.411196 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:39:26.411229 3798194 memory.go:191] Add success. I1121 10:39:26.411561 3798194 cpu.go:282] Add success. I1121 10:39:26.442209 3798194 net.go:642] Add success. I1121 10:39:26.446096 3798194 net.go:778] primary dev: ETH0 I1121 10:39:26.446127 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:39:26.446142 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:39:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:39:29.450124 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:39:29.450153 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:39:36.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:39:36.412250 3798194 memory.go:184] no items to output this cycle I1121 10:39:36.412395 3798194 cpu.go:275] no items to output this cycle E1121 10:39:46.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:39:46.412268 3798194 memory.go:184] no items to output this cycle I1121 10:39:46.412414 3798194 cpu.go:275] no items to output this cycle E1121 10:39:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:39:56.411952 3798194 memory.go:191] Add success. W1121 10:39:56.411981 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:39:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:39:56.411996 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:39:56.412380 3798194 cpu.go:282] Add success. I1121 10:39:56.441821 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:39:56.442336 3798194 net.go:642] Add success. I1121 10:39:56.446243 3798194 net.go:778] primary dev: ETH0 I1121 10:39:56.446259 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:39:56.446273 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:39:56.504124 3798194 disk_info.go:125] begin check local disk info of client I1121 10:39:56.507662 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:39:56.507671 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037c880 0xc00037c8c0] I1121 10:39:57.450564 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:39:57.450722 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:39:57.450774 3798194 disk_worker.go:723] disk space is not compliant W1121 10:39:57.450777 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:39:57.459193 3798194 disk_worker.go:509] system disk:vda1 I1121 10:39:57.459234 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:39:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:39:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:39:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:39:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:39:59.471674 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:40:06.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:40:06.413264 3798194 memory.go:184] no items to output this cycle I1121 10:40:06.413410 3798194 cpu.go:275] no items to output this cycle E1121 10:40:16.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:40:16.412263 3798194 memory.go:184] no items to output this cycle I1121 10:40:16.412409 3798194 cpu.go:275] no items to output this cycle E1121 10:40:26.412931 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:40:26.412966 3798194 memory.go:191] Add success. I1121 10:40:26.417265 3798194 cpu.go:282] Add success. I1121 10:40:26.441735 3798194 net.go:642] Add success. I1121 10:40:26.443390 3798194 net.go:778] primary dev: ETH0 I1121 10:40:26.443404 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:40:26.443421 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:40:29.450611 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:40:29.450698 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:40:29.450722 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:40:36.413004 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:40:36.413028 3798194 memory.go:184] no items to output this cycle I1121 10:40:36.413193 3798194 cpu.go:275] no items to output this cycle E1121 10:40:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:40:46.411980 3798194 memory.go:184] no items to output this cycle I1121 10:40:46.412144 3798194 cpu.go:275] no items to output this cycle E1121 10:40:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:40:56.411953 3798194 memory.go:191] Add success. W1121 10:40:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:40:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:40:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:40:56.412349 3798194 cpu.go:282] Add success. I1121 10:40:56.444673 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:40:56.445295 3798194 net.go:642] Add success. I1121 10:40:56.449208 3798194 net.go:778] primary dev: ETH0 I1121 10:40:56.449225 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:40:56.449240 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:40:56.508308 3798194 disk_info.go:125] begin check local disk info of client I1121 10:40:56.512082 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:40:56.512090 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000441e40 0xc000441e80] E1121 10:40:57.450991 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 10:40:57.451005 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:40:57.451262 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:40:57.451271 3798194 disk_worker.go:723] disk space is not compliant W1121 10:40:57.451274 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:40:57.459372 3798194 disk_worker.go:509] system disk:vda1 I1121 10:40:57.459409 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 10:40:58.447997 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 10:40:58.448011 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 10:40:59.448621 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 10:40:59.449695 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:40:59.449745 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:40:59.449764 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:40:59.471283 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:41:06.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:41:06.412258 3798194 memory.go:184] no items to output this cycle I1121 10:41:06.412416 3798194 cpu.go:275] no items to output this cycle E1121 10:41:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:41:16.411979 3798194 memory.go:184] no items to output this cycle I1121 10:41:16.412138 3798194 cpu.go:275] no items to output this cycle E1121 10:41:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:41:26.411954 3798194 memory.go:191] Add success. I1121 10:41:26.412297 3798194 cpu.go:282] Add success. I1121 10:41:26.442054 3798194 net.go:642] Add success. I1121 10:41:26.445994 3798194 net.go:778] primary dev: ETH0 I1121 10:41:26.446009 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:41:26.446022 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:41:29.450403 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:41:29.450496 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:41:29.450524 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:41:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:41:36.412253 3798194 memory.go:184] no items to output this cycle I1121 10:41:36.412395 3798194 cpu.go:275] no items to output this cycle E1121 10:41:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:41:46.412257 3798194 memory.go:184] no items to output this cycle I1121 10:41:46.412403 3798194 cpu.go:275] no items to output this cycle E1121 10:41:56.411929 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:41:56.411960 3798194 memory.go:191] Add success. W1121 10:41:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:41:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:41:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:41:56.412358 3798194 cpu.go:282] Add success. I1121 10:41:56.441795 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:41:56.442314 3798194 net.go:642] Add success. I1121 10:41:56.446451 3798194 net.go:778] primary dev: ETH0 I1121 10:41:56.446467 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:41:56.446479 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:41:56.512200 3798194 disk_info.go:125] begin check local disk info of client I1121 10:41:56.516025 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:41:56.516034 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b53c0 0xc0003b5400] I1121 10:41:57.450604 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:41:57.450818 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:41:57.450830 3798194 disk_worker.go:723] disk space is not compliant W1121 10:41:57.450833 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:41:57.459286 3798194 disk_worker.go:509] system disk:vda1 I1121 10:41:57.459326 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:41:58.445958 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:41:59.450067 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:41:59.450153 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:41:59.450184 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:41:59.471684 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:42:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:42:06.412257 3798194 memory.go:184] no items to output this cycle I1121 10:42:06.412400 3798194 cpu.go:275] no items to output this cycle E1121 10:42:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:42:16.411986 3798194 memory.go:184] no items to output this cycle I1121 10:42:16.412355 3798194 cpu.go:275] no items to output this cycle I1121 10:42:17.440039 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 10:42:17.440047 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 10:42:26.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:42:26.413264 3798194 memory.go:191] Add success. I1121 10:42:26.417597 3798194 cpu.go:282] Add success. I1121 10:42:26.442123 3798194 net.go:642] Add success. I1121 10:42:26.443536 3798194 net.go:778] primary dev: ETH0 I1121 10:42:26.443550 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:42:26.443562 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:42:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:42:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:42:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:42:36.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:42:36.411984 3798194 memory.go:184] no items to output this cycle I1121 10:42:36.412142 3798194 cpu.go:275] no items to output this cycle E1121 10:42:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:42:46.411973 3798194 memory.go:184] no items to output this cycle I1121 10:42:46.412126 3798194 cpu.go:275] no items to output this cycle E1121 10:42:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:42:56.411951 3798194 memory.go:191] Add success. W1121 10:42:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:42:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:42:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:42:56.412354 3798194 cpu.go:282] Add success. I1121 10:42:56.441841 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:42:56.442350 3798194 net.go:642] Add success. I1121 10:42:56.446321 3798194 net.go:778] primary dev: ETH0 I1121 10:42:56.446336 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:42:56.446351 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:42:56.516139 3798194 disk_info.go:125] begin check local disk info of client I1121 10:42:56.520049 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:42:56.520057 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002ea380 0xc0002ea3c0] I1121 10:42:57.450649 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:42:57.450812 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:42:57.450867 3798194 disk_worker.go:723] disk space is not compliant W1121 10:42:57.450871 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:42:57.459175 3798194 disk_worker.go:509] system disk:vda1 I1121 10:42:57.459215 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:42:58.445995 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:42:59.450373 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:42:59.450449 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:42:59.450474 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:42:59.471004 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:43:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:43:06.412261 3798194 memory.go:184] no items to output this cycle I1121 10:43:06.412403 3798194 cpu.go:275] no items to output this cycle E1121 10:43:16.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:43:16.413268 3798194 memory.go:184] no items to output this cycle I1121 10:43:16.413420 3798194 cpu.go:275] no items to output this cycle E1121 10:43:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:43:26.411957 3798194 memory.go:191] Add success. I1121 10:43:26.412284 3798194 cpu.go:282] Add success. I1121 10:43:26.441741 3798194 net.go:642] Add success. I1121 10:43:26.445799 3798194 net.go:778] primary dev: ETH0 I1121 10:43:26.445813 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:43:26.445826 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:43:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:43:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:43:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:43:36.412766 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:43:36.412787 3798194 memory.go:184] no items to output this cycle I1121 10:43:36.412932 3798194 cpu.go:275] no items to output this cycle E1121 10:43:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:43:46.413265 3798194 memory.go:184] no items to output this cycle I1121 10:43:46.413412 3798194 cpu.go:275] no items to output this cycle E1121 10:43:56.411932 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:43:56.411966 3798194 memory.go:191] Add success. W1121 10:43:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:43:56.412010 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:43:56.412014 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:43:56.412397 3798194 cpu.go:282] Add success. I1121 10:43:56.441940 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:43:56.442541 3798194 net.go:642] Add success. I1121 10:43:56.446802 3798194 net.go:778] primary dev: ETH0 I1121 10:43:56.446817 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:43:56.446831 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:43:56.520931 3798194 disk_info.go:125] begin check local disk info of client I1121 10:43:56.524895 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:43:56.524904 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000364500 0xc000364540] I1121 10:43:57.450617 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:43:57.450823 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:43:57.450833 3798194 disk_worker.go:723] disk space is not compliant W1121 10:43:57.450837 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:43:57.459239 3798194 disk_worker.go:509] system disk:vda1 I1121 10:43:57.459281 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:43:58.446014 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:43:59.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:43:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:43:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:43:59.471673 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:44:06.413163 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:44:06.413189 3798194 memory.go:184] no items to output this cycle I1121 10:44:06.413334 3798194 cpu.go:275] no items to output this cycle E1121 10:44:16.412329 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:44:16.412349 3798194 memory.go:184] no items to output this cycle I1121 10:44:16.412496 3798194 cpu.go:275] no items to output this cycle E1121 10:44:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:44:26.413266 3798194 memory.go:191] Add success. I1121 10:44:26.417548 3798194 cpu.go:282] Add success. I1121 10:44:26.442084 3798194 net.go:642] Add success. I1121 10:44:26.443566 3798194 net.go:778] primary dev: ETH0 I1121 10:44:26.443580 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:44:26.443596 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:44:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:44:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:44:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:44:36.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:44:36.412881 3798194 memory.go:184] no items to output this cycle I1121 10:44:36.413026 3798194 cpu.go:275] no items to output this cycle E1121 10:44:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:44:46.411981 3798194 memory.go:184] no items to output this cycle I1121 10:44:46.412146 3798194 cpu.go:275] no items to output this cycle E1121 10:44:56.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:44:56.413272 3798194 memory.go:191] Add success. W1121 10:44:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:44:56.413314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:44:56.413317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:44:56.417589 3798194 cpu.go:282] Add success. I1121 10:44:56.440908 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:44:56.441490 3798194 net.go:642] Add success. I1121 10:44:56.443217 3798194 net.go:778] primary dev: ETH0 I1121 10:44:56.443233 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:44:56.443246 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:44:56.525001 3798194 disk_info.go:125] begin check local disk info of client I1121 10:44:56.528951 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:44:56.528961 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000461e00 0xc000461e40] I1121 10:44:57.446414 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:44:57.446636 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:44:57.446702 3798194 disk_worker.go:723] disk space is not compliant W1121 10:44:57.446705 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:44:57.450310 3798194 disk_worker.go:509] system disk:vda1 I1121 10:44:57.450347 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:44:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:44:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:44:59.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:44:59.450515 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:44:59.471057 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:45:06.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:45:06.412263 3798194 memory.go:184] no items to output this cycle I1121 10:45:06.412412 3798194 cpu.go:275] no items to output this cycle E1121 10:45:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:45:16.411974 3798194 memory.go:184] no items to output this cycle I1121 10:45:16.412130 3798194 cpu.go:275] no items to output this cycle I1121 10:45:17.441039 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 10:45:17.441047 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 10:45:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:45:26.413267 3798194 memory.go:191] Add success. I1121 10:45:26.417476 3798194 cpu.go:282] Add success. I1121 10:45:26.442014 3798194 net.go:642] Add success. I1121 10:45:26.443424 3798194 net.go:778] primary dev: ETH0 I1121 10:45:26.443440 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:45:26.443453 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:45:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:45:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:45:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:45:36.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:45:36.413267 3798194 memory.go:184] no items to output this cycle I1121 10:45:36.413412 3798194 cpu.go:275] no items to output this cycle E1121 10:45:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:45:46.413255 3798194 memory.go:184] no items to output this cycle I1121 10:45:46.413401 3798194 cpu.go:275] no items to output this cycle E1121 10:45:56.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:45:56.412262 3798194 memory.go:191] Add success. W1121 10:45:56.412294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:45:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:45:56.412309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:45:56.416864 3798194 cpu.go:282] Add success. I1121 10:45:56.443919 3798194 event_worker.go:152] Polling the log file for events... I1121 10:45:56.444160 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:45:56.444630 3798194 net.go:642] Add success. I1121 10:45:56.446172 3798194 net.go:778] primary dev: ETH0 I1121 10:45:56.446186 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:45:56.446197 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:45:56.529063 3798194 disk_info.go:125] begin check local disk info of client I1121 10:45:56.532942 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:45:56.532951 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a9300 0xc0003a9340] E1121 10:45:57.450567 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 10:45:57.450580 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:45:57.450781 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:45:57.450792 3798194 disk_worker.go:723] disk space is not compliant W1121 10:45:57.450796 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:45:57.464234 3798194 disk_worker.go:509] system disk:vda1 I1121 10:45:57.464284 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 10:45:58.448497 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 10:45:58.448513 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 10:45:59.449147 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 10:45:59.450221 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:45:59.450270 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:45:59.450289 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:45:59.470727 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:46:06.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:46:06.411969 3798194 memory.go:184] no items to output this cycle I1121 10:46:06.412108 3798194 cpu.go:275] no items to output this cycle E1121 10:46:16.412766 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:46:16.412800 3798194 memory.go:184] no items to output this cycle I1121 10:46:16.412943 3798194 cpu.go:275] no items to output this cycle E1121 10:46:26.411364 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:46:26.411403 3798194 memory.go:191] Add success. I1121 10:46:26.415522 3798194 cpu.go:282] Add success. I1121 10:46:26.442071 3798194 net.go:642] Add success. I1121 10:46:26.443483 3798194 net.go:778] primary dev: ETH0 I1121 10:46:26.443498 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:46:26.443511 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:46:29.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:46:29.450126 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:46:29.450158 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:46:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:46:36.411975 3798194 memory.go:184] no items to output this cycle I1121 10:46:36.412130 3798194 cpu.go:275] no items to output this cycle E1121 10:46:46.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:46:46.413252 3798194 memory.go:184] no items to output this cycle I1121 10:46:46.413400 3798194 cpu.go:275] no items to output this cycle E1121 10:46:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:46:56.411947 3798194 memory.go:191] Add success. W1121 10:46:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:46:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:46:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:46:56.412350 3798194 cpu.go:282] Add success. I1121 10:46:56.441846 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:46:56.442498 3798194 net.go:642] Add success. I1121 10:46:56.446608 3798194 net.go:778] primary dev: ETH0 I1121 10:46:56.446623 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:46:56.446634 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:46:56.533786 3798194 disk_info.go:125] begin check local disk info of client I1121 10:46:56.537674 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:46:56.537683 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000265a80 0xc000265ac0] I1121 10:46:57.450603 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:46:57.450763 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:46:57.450814 3798194 disk_worker.go:723] disk space is not compliant W1121 10:46:57.450817 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:46:57.459086 3798194 disk_worker.go:509] system disk:vda1 I1121 10:46:57.459140 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:46:58.446174 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:46:59.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:46:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:46:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:46:59.471722 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:47:06.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:47:06.411987 3798194 memory.go:184] no items to output this cycle I1121 10:47:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 10:47:16.413406 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:47:16.413427 3798194 memory.go:184] no items to output this cycle I1121 10:47:16.413570 3798194 cpu.go:275] no items to output this cycle E1121 10:47:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:47:26.411954 3798194 memory.go:191] Add success. I1121 10:47:26.412303 3798194 cpu.go:282] Add success. I1121 10:47:26.442064 3798194 net.go:642] Add success. I1121 10:47:26.446277 3798194 net.go:778] primary dev: ETH0 I1121 10:47:26.446295 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:47:26.446308 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:47:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:47:29.450488 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:47:29.450515 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:47:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:47:36.411980 3798194 memory.go:184] no items to output this cycle I1121 10:47:36.412144 3798194 cpu.go:275] no items to output this cycle E1121 10:47:46.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:47:46.412256 3798194 memory.go:184] no items to output this cycle I1121 10:47:46.412399 3798194 cpu.go:275] no items to output this cycle E1121 10:47:56.412028 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:47:56.412159 3798194 memory.go:191] Add success. W1121 10:47:56.412192 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:47:56.412204 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:47:56.412208 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:47:56.412907 3798194 cpu.go:282] Add success. I1121 10:47:56.440965 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:47:56.441541 3798194 net.go:642] Add success. I1121 10:47:56.446042 3798194 net.go:778] primary dev: ETH0 I1121 10:47:56.446062 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:47:56.446081 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:47:56.538347 3798194 disk_info.go:125] begin check local disk info of client I1121 10:47:56.542188 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:47:56.542200 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071140 0xc000071180] I1121 10:47:57.450509 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:47:57.450676 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:47:57.450728 3798194 disk_worker.go:723] disk space is not compliant W1121 10:47:57.450732 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:47:57.459156 3798194 disk_worker.go:509] system disk:vda1 I1121 10:47:57.459220 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:47:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:47:59.450434 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:47:59.450519 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:47:59.450553 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:47:59.471242 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:48:06.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:48:06.411994 3798194 memory.go:184] no items to output this cycle I1121 10:48:06.412156 3798194 cpu.go:275] no items to output this cycle E1121 10:48:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:48:16.412259 3798194 memory.go:184] no items to output this cycle I1121 10:48:16.412413 3798194 cpu.go:275] no items to output this cycle I1121 10:48:17.441759 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 10:48:17.441768 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 10:48:26.411330 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:48:26.411362 3798194 memory.go:191] Add success. I1121 10:48:26.411698 3798194 cpu.go:282] Add success. I1121 10:48:26.441459 3798194 net.go:642] Add success. I1121 10:48:26.445260 3798194 net.go:778] primary dev: ETH0 I1121 10:48:26.445275 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:48:26.445287 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:48:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:48:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:48:29.450494 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:48:36.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:48:36.411983 3798194 memory.go:184] no items to output this cycle I1121 10:48:36.412140 3798194 cpu.go:275] no items to output this cycle E1121 10:48:46.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:48:46.412879 3798194 memory.go:184] no items to output this cycle I1121 10:48:46.413024 3798194 cpu.go:275] no items to output this cycle E1121 10:48:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:48:56.411956 3798194 memory.go:191] Add success. W1121 10:48:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:48:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:48:56.412007 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:48:56.412361 3798194 cpu.go:282] Add success. I1121 10:48:56.441941 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:48:56.442526 3798194 net.go:642] Add success. I1121 10:48:56.446840 3798194 net.go:778] primary dev: ETH0 I1121 10:48:56.446856 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:48:56.446872 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:48:56.543072 3798194 disk_info.go:125] begin check local disk info of client I1121 10:48:56.546713 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:48:56.546722 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037e780 0xc00037e7c0] I1121 10:48:57.450186 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:48:57.450353 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:48:57.450411 3798194 disk_worker.go:723] disk space is not compliant W1121 10:48:57.450414 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:48:57.458403 3798194 disk_worker.go:509] system disk:vda1 I1121 10:48:57.458449 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:48:58.446133 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:48:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:48:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:48:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:48:59.471644 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:49:06.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:49:06.411980 3798194 memory.go:184] no items to output this cycle I1121 10:49:06.412144 3798194 cpu.go:275] no items to output this cycle E1121 10:49:16.412365 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:49:16.412386 3798194 memory.go:184] no items to output this cycle I1121 10:49:16.412543 3798194 cpu.go:275] no items to output this cycle E1121 10:49:26.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:49:26.411960 3798194 memory.go:191] Add success. I1121 10:49:26.412305 3798194 cpu.go:282] Add success. I1121 10:49:26.441943 3798194 net.go:642] Add success. I1121 10:49:26.446015 3798194 net.go:778] primary dev: ETH0 I1121 10:49:26.446029 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:49:26.446043 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:49:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:49:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:49:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:49:36.412098 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:49:36.412142 3798194 memory.go:184] no items to output this cycle I1121 10:49:36.412299 3798194 cpu.go:275] no items to output this cycle E1121 10:49:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:49:46.411982 3798194 memory.go:184] no items to output this cycle I1121 10:49:46.412148 3798194 cpu.go:275] no items to output this cycle E1121 10:49:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:49:56.411949 3798194 memory.go:191] Add success. W1121 10:49:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:49:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:49:56.411993 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:49:56.412354 3798194 cpu.go:282] Add success. I1121 10:49:56.441808 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:49:56.442323 3798194 net.go:642] Add success. I1121 10:49:56.446657 3798194 net.go:778] primary dev: ETH0 I1121 10:49:56.446674 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:49:56.446688 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:49:56.546821 3798194 disk_info.go:125] begin check local disk info of client I1121 10:49:56.550469 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:49:56.550478 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003867c0 0xc000386800] I1121 10:49:57.450386 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:49:57.450593 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:49:57.450603 3798194 disk_worker.go:723] disk space is not compliant W1121 10:49:57.450607 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:49:57.458692 3798194 disk_worker.go:509] system disk:vda1 I1121 10:49:57.458731 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:49:58.446486 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:49:59.450592 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:49:59.450681 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:49:59.450707 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:49:59.471258 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:50:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:50:06.412257 3798194 memory.go:184] no items to output this cycle I1121 10:50:06.412407 3798194 cpu.go:275] no items to output this cycle E1121 10:50:16.411972 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:50:16.411994 3798194 memory.go:184] no items to output this cycle I1121 10:50:16.412154 3798194 cpu.go:275] no items to output this cycle E1121 10:50:26.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:50:26.411950 3798194 memory.go:191] Add success. I1121 10:50:26.412304 3798194 cpu.go:282] Add success. I1121 10:50:26.441776 3798194 net.go:642] Add success. I1121 10:50:26.446081 3798194 net.go:778] primary dev: ETH0 I1121 10:50:26.446097 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:50:26.446128 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:50:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:50:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:50:29.450165 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:50:36.413123 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:50:36.413148 3798194 memory.go:184] no items to output this cycle I1121 10:50:36.413307 3798194 cpu.go:275] no items to output this cycle E1121 10:50:46.413310 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:50:46.413333 3798194 memory.go:184] no items to output this cycle I1121 10:50:46.413489 3798194 cpu.go:275] no items to output this cycle E1121 10:50:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:50:56.411949 3798194 memory.go:191] Add success. W1121 10:50:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:50:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:50:56.411990 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:50:56.412350 3798194 cpu.go:282] Add success. I1121 10:50:56.444355 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:50:56.444940 3798194 net.go:642] Add success. I1121 10:50:56.449227 3798194 net.go:778] primary dev: ETH0 I1121 10:50:56.449243 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:50:56.449257 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:50:56.551503 3798194 disk_info.go:125] begin check local disk info of client I1121 10:50:56.554993 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:50:56.555002 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000386440 0xc000386480] E1121 10:50:57.450968 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 10:50:57.450981 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:50:57.451213 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:50:57.451223 3798194 disk_worker.go:723] disk space is not compliant W1121 10:50:57.451226 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:50:57.460016 3798194 disk_worker.go:509] system disk:vda1 I1121 10:50:57.460059 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 10:50:58.448775 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 10:50:58.448788 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 10:50:59.448991 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 10:50:59.450067 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:50:59.450130 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:50:59.450149 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:50:59.471671 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:51:06.412775 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:51:06.412796 3798194 memory.go:184] no items to output this cycle I1121 10:51:06.412960 3798194 cpu.go:275] no items to output this cycle E1121 10:51:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:51:16.411984 3798194 memory.go:184] no items to output this cycle I1121 10:51:16.412138 3798194 cpu.go:275] no items to output this cycle I1121 10:51:17.442436 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 10:51:17.442447 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 10:51:26.411386 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:51:26.411420 3798194 memory.go:191] Add success. I1121 10:51:26.411740 3798194 cpu.go:282] Add success. I1121 10:51:26.441305 3798194 net.go:642] Add success. I1121 10:51:26.445553 3798194 net.go:778] primary dev: ETH0 I1121 10:51:26.445569 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:51:26.445583 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:51:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:51:29.450151 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:51:29.450185 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:51:36.412809 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:51:36.412831 3798194 memory.go:184] no items to output this cycle I1121 10:51:36.412972 3798194 cpu.go:275] no items to output this cycle E1121 10:51:46.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:51:46.412881 3798194 memory.go:184] no items to output this cycle I1121 10:51:46.413027 3798194 cpu.go:275] no items to output this cycle E1121 10:51:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:51:56.412274 3798194 memory.go:191] Add success. W1121 10:51:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:51:56.412318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:51:56.412322 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:51:56.416706 3798194 cpu.go:282] Add success. I1121 10:51:56.441089 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:51:56.441581 3798194 net.go:642] Add success. I1121 10:51:56.442945 3798194 net.go:778] primary dev: ETH0 I1121 10:51:56.442967 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:51:56.442981 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:51:56.555143 3798194 disk_info.go:125] begin check local disk info of client I1121 10:51:56.559369 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:51:56.559378 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037f4c0 0xc00037f500] I1121 10:51:57.450983 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:51:57.451225 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:51:57.451237 3798194 disk_worker.go:723] disk space is not compliant W1121 10:51:57.451240 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:51:57.460045 3798194 disk_worker.go:509] system disk:vda1 I1121 10:51:57.460091 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:51:58.446377 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:51:59.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:51:59.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:51:59.450494 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:51:59.471026 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:52:06.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:52:06.411974 3798194 memory.go:184] no items to output this cycle I1121 10:52:06.412134 3798194 cpu.go:275] no items to output this cycle E1121 10:52:16.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:52:16.412267 3798194 memory.go:184] no items to output this cycle I1121 10:52:16.412412 3798194 cpu.go:275] no items to output this cycle E1121 10:52:26.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:52:26.412884 3798194 memory.go:191] Add success. I1121 10:52:26.419450 3798194 cpu.go:282] Add success. I1121 10:52:26.442074 3798194 net.go:642] Add success. I1121 10:52:26.443564 3798194 net.go:778] primary dev: ETH0 I1121 10:52:26.443578 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:52:26.443589 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:52:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:52:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:52:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:52:36.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:52:36.413265 3798194 memory.go:184] no items to output this cycle I1121 10:52:36.413412 3798194 cpu.go:275] no items to output this cycle E1121 10:52:46.412330 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:52:46.412352 3798194 memory.go:184] no items to output this cycle I1121 10:52:46.412500 3798194 cpu.go:275] no items to output this cycle E1121 10:52:56.411930 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:52:56.411963 3798194 memory.go:191] Add success. W1121 10:52:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:52:56.412008 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:52:56.412011 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:52:56.412353 3798194 cpu.go:282] Add success. I1121 10:52:56.441939 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:52:56.442554 3798194 net.go:642] Add success. I1121 10:52:56.447229 3798194 net.go:778] primary dev: ETH0 I1121 10:52:56.447244 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:52:56.447261 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:52:56.559470 3798194 disk_info.go:125] begin check local disk info of client I1121 10:52:56.563444 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:52:56.563453 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000468fc0 0xc000469000] I1121 10:52:57.450595 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:52:57.450770 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:52:57.450832 3798194 disk_worker.go:723] disk space is not compliant W1121 10:52:57.450836 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:52:57.459939 3798194 disk_worker.go:509] system disk:vda1 I1121 10:52:57.459981 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:52:58.446012 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:52:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:52:59.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:52:59.450127 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:52:59.471666 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:53:06.411980 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:53:06.412010 3798194 memory.go:184] no items to output this cycle I1121 10:53:06.412173 3798194 cpu.go:275] no items to output this cycle E1121 10:53:16.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:53:16.413270 3798194 memory.go:184] no items to output this cycle I1121 10:53:16.413417 3798194 cpu.go:275] no items to output this cycle E1121 10:53:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:53:26.413269 3798194 memory.go:191] Add success. I1121 10:53:26.418300 3798194 cpu.go:282] Add success. I1121 10:53:26.441830 3798194 net.go:642] Add success. I1121 10:53:26.443341 3798194 net.go:778] primary dev: ETH0 I1121 10:53:26.443359 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:53:26.443374 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:53:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:53:29.450494 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:53:29.450523 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:53:36.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:53:36.411974 3798194 memory.go:184] no items to output this cycle I1121 10:53:36.412136 3798194 cpu.go:275] no items to output this cycle E1121 10:53:46.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:53:46.412252 3798194 memory.go:184] no items to output this cycle I1121 10:53:46.412400 3798194 cpu.go:275] no items to output this cycle E1121 10:53:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:53:56.411957 3798194 memory.go:191] Add success. W1121 10:53:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:53:56.412001 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:53:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:53:56.412342 3798194 cpu.go:282] Add success. I1121 10:53:56.441791 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:53:56.442326 3798194 net.go:642] Add success. I1121 10:53:56.446645 3798194 net.go:778] primary dev: ETH0 I1121 10:53:56.446659 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:53:56.446671 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:53:56.563552 3798194 disk_info.go:125] begin check local disk info of client I1121 10:53:56.567506 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:53:56.567515 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037d140 0xc00037d180] I1121 10:53:57.450619 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:53:57.450796 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:53:57.450852 3798194 disk_worker.go:723] disk space is not compliant W1121 10:53:57.450855 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:53:57.460680 3798194 disk_worker.go:509] system disk:vda1 I1121 10:53:57.460729 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:53:58.446010 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:53:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:53:59.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:53:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:53:59.471653 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:54:06.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:54:06.413271 3798194 memory.go:184] no items to output this cycle I1121 10:54:06.413415 3798194 cpu.go:275] no items to output this cycle E1121 10:54:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:54:16.411981 3798194 memory.go:184] no items to output this cycle I1121 10:54:16.412134 3798194 cpu.go:275] no items to output this cycle I1121 10:54:17.442568 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 10:54:17.442584 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 10:54:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:54:26.413270 3798194 memory.go:191] Add success. I1121 10:54:26.417682 3798194 cpu.go:282] Add success. I1121 10:54:26.441245 3798194 net.go:642] Add success. I1121 10:54:26.447760 3798194 net.go:778] primary dev: ETH0 I1121 10:54:26.447777 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:54:26.447791 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:54:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:54:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:54:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:54:36.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:54:36.413249 3798194 memory.go:184] no items to output this cycle I1121 10:54:36.413393 3798194 cpu.go:275] no items to output this cycle E1121 10:54:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:54:46.411976 3798194 memory.go:184] no items to output this cycle I1121 10:54:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 10:54:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:54:56.411949 3798194 memory.go:191] Add success. W1121 10:54:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:54:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:54:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:54:56.412356 3798194 cpu.go:282] Add success. I1121 10:54:56.441836 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:54:56.442404 3798194 net.go:642] Add success. I1121 10:54:56.446227 3798194 net.go:778] primary dev: ETH0 I1121 10:54:56.446241 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:54:56.446261 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:54:56.569218 3798194 disk_info.go:125] begin check local disk info of client I1121 10:54:56.572807 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:54:56.572816 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002f5a80 0xc0002f5ac0] I1121 10:54:57.450369 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:54:57.450580 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:54:57.450591 3798194 disk_worker.go:723] disk space is not compliant W1121 10:54:57.450594 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:54:57.458715 3798194 disk_worker.go:509] system disk:vda1 I1121 10:54:57.458756 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:54:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:54:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:54:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:54:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:54:59.471706 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:55:06.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:55:06.413260 3798194 memory.go:184] no items to output this cycle I1121 10:55:06.413403 3798194 cpu.go:275] no items to output this cycle E1121 10:55:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:55:16.411976 3798194 memory.go:184] no items to output this cycle I1121 10:55:16.412141 3798194 cpu.go:275] no items to output this cycle E1121 10:55:26.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:55:26.413258 3798194 memory.go:191] Add success. I1121 10:55:26.417331 3798194 cpu.go:282] Add success. I1121 10:55:26.441996 3798194 net.go:642] Add success. I1121 10:55:26.443508 3798194 net.go:778] primary dev: ETH0 I1121 10:55:26.443523 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:55:26.443535 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:55:29.450408 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:55:29.450488 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:55:29.450514 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:55:36.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:55:36.411990 3798194 memory.go:184] no items to output this cycle I1121 10:55:36.412155 3798194 cpu.go:275] no items to output this cycle E1121 10:55:46.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:55:46.413252 3798194 memory.go:184] no items to output this cycle I1121 10:55:46.413600 3798194 cpu.go:275] no items to output this cycle E1121 10:55:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:55:56.411947 3798194 memory.go:191] Add success. W1121 10:55:56.411974 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:55:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:55:56.411989 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:55:56.412349 3798194 cpu.go:282] Add success. I1121 10:55:56.441695 3798194 event_worker.go:152] Polling the log file for events... I1121 10:55:56.444855 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:55:56.445351 3798194 net.go:642] Add success. I1121 10:55:56.449452 3798194 net.go:778] primary dev: ETH0 I1121 10:55:56.449468 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:55:56.449480 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:55:56.573810 3798194 disk_info.go:125] begin check local disk info of client I1121 10:55:56.577641 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:55:56.577651 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e1400 0xc0002e1440] E1121 10:55:57.450303 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 10:55:57.450318 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:55:57.450514 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:55:57.450523 3798194 disk_worker.go:723] disk space is not compliant W1121 10:55:57.450526 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:55:57.458630 3798194 disk_worker.go:509] system disk:vda1 I1121 10:55:57.458673 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 10:55:58.447817 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 10:55:58.447833 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 10:55:59.448523 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 10:55:59.450652 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:55:59.450702 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:55:59.450721 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:55:59.471246 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:56:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:56:06.411977 3798194 memory.go:184] no items to output this cycle I1121 10:56:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 10:56:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:56:16.411974 3798194 memory.go:184] no items to output this cycle I1121 10:56:16.412132 3798194 cpu.go:275] no items to output this cycle E1121 10:56:26.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:56:26.412261 3798194 memory.go:191] Add success. I1121 10:56:26.417294 3798194 cpu.go:282] Add success. I1121 10:56:26.441985 3798194 net.go:642] Add success. I1121 10:56:26.443545 3798194 net.go:778] primary dev: ETH0 I1121 10:56:26.443561 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:56:26.443572 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:56:29.450037 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:56:29.450137 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:56:29.450167 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:56:36.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:56:36.412889 3798194 memory.go:184] no items to output this cycle I1121 10:56:36.413047 3798194 cpu.go:275] no items to output this cycle E1121 10:56:46.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:56:46.412267 3798194 memory.go:184] no items to output this cycle I1121 10:56:46.412421 3798194 cpu.go:275] no items to output this cycle E1121 10:56:56.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:56:56.411983 3798194 memory.go:191] Add success. W1121 10:56:56.412018 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:56:56.412031 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:56:56.412035 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:56:56.412400 3798194 cpu.go:282] Add success. I1121 10:56:56.441127 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:56:56.441624 3798194 net.go:642] Add success. I1121 10:56:56.445823 3798194 net.go:778] primary dev: ETH0 I1121 10:56:56.445838 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:56:56.445850 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:56:56.577746 3798194 disk_info.go:125] begin check local disk info of client I1121 10:56:56.582256 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:56:56.582266 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00042d940 0xc00042d980] I1121 10:56:57.451409 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:56:57.451617 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:56:57.451629 3798194 disk_worker.go:723] disk space is not compliant W1121 10:56:57.451632 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:56:57.464359 3798194 disk_worker.go:509] system disk:vda1 I1121 10:56:57.464411 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:56:58.445986 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:56:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:56:59.450128 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:56:59.450155 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:56:59.471718 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:57:06.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:57:06.411988 3798194 memory.go:184] no items to output this cycle I1121 10:57:06.412145 3798194 cpu.go:275] no items to output this cycle E1121 10:57:16.412845 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:57:16.412868 3798194 memory.go:184] no items to output this cycle I1121 10:57:16.413017 3798194 cpu.go:275] no items to output this cycle I1121 10:57:17.443426 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 10:57:17.443434 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 10:57:26.411239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:57:26.411275 3798194 memory.go:191] Add success. I1121 10:57:26.411591 3798194 cpu.go:282] Add success. I1121 10:57:26.442211 3798194 net.go:642] Add success. I1121 10:57:26.446264 3798194 net.go:778] primary dev: ETH0 I1121 10:57:26.446280 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:57:26.446293 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:57:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:57:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:57:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:57:36.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:57:36.413259 3798194 memory.go:184] no items to output this cycle I1121 10:57:36.413401 3798194 cpu.go:275] no items to output this cycle E1121 10:57:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:57:46.411979 3798194 memory.go:184] no items to output this cycle I1121 10:57:46.412143 3798194 cpu.go:275] no items to output this cycle E1121 10:57:56.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:57:56.413270 3798194 memory.go:191] Add success. W1121 10:57:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:57:56.413315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:57:56.413318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:57:56.417310 3798194 cpu.go:282] Add success. I1121 10:57:56.441680 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:57:56.442272 3798194 net.go:642] Add success. I1121 10:57:56.443647 3798194 net.go:778] primary dev: ETH0 I1121 10:57:56.443661 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:57:56.443673 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:57:56.583120 3798194 disk_info.go:125] begin check local disk info of client I1121 10:57:56.587034 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:57:56.587046 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000398900 0xc000398940] I1121 10:57:57.450291 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:57:57.450502 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:57:57.450514 3798194 disk_worker.go:723] disk space is not compliant W1121 10:57:57.450517 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:57:57.458658 3798194 disk_worker.go:509] system disk:vda1 I1121 10:57:57.458699 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:57:58.446389 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:57:59.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:57:59.450119 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:57:59.450147 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:57:59.471660 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:58:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:58:06.411984 3798194 memory.go:184] no items to output this cycle I1121 10:58:06.412137 3798194 cpu.go:275] no items to output this cycle E1121 10:58:16.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:58:16.412251 3798194 memory.go:184] no items to output this cycle I1121 10:58:16.412402 3798194 cpu.go:275] no items to output this cycle E1121 10:58:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:58:26.411952 3798194 memory.go:191] Add success. I1121 10:58:26.412307 3798194 cpu.go:282] Add success. I1121 10:58:26.441316 3798194 net.go:642] Add success. I1121 10:58:26.445272 3798194 net.go:778] primary dev: ETH0 I1121 10:58:26.445288 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:58:26.445301 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:58:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:58:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:58:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:58:36.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:58:36.412873 3798194 memory.go:184] no items to output this cycle I1121 10:58:36.413033 3798194 cpu.go:275] no items to output this cycle E1121 10:58:46.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:58:46.411990 3798194 memory.go:184] no items to output this cycle I1121 10:58:46.412150 3798194 cpu.go:275] no items to output this cycle E1121 10:58:56.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:58:56.412893 3798194 memory.go:191] Add success. W1121 10:58:56.412926 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:58:56.412938 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:58:56.412941 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:58:56.416996 3798194 cpu.go:282] Add success. I1121 10:58:56.441382 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:58:56.441953 3798194 net.go:642] Add success. I1121 10:58:56.443446 3798194 net.go:778] primary dev: ETH0 I1121 10:58:56.443460 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:58:56.443474 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:58:56.587946 3798194 disk_info.go:125] begin check local disk info of client I1121 10:58:56.591875 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:58:56.591883 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00025c4c0 0xc00025c500] I1121 10:58:57.450242 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:58:57.450464 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:58:57.450475 3798194 disk_worker.go:723] disk space is not compliant W1121 10:58:57.450478 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:58:57.458542 3798194 disk_worker.go:509] system disk:vda1 I1121 10:58:57.458583 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:58:58.445996 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:58:59.450483 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:58:59.450564 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:58:59.450589 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:58:59.471121 3798194 disk_local_worker.go:436] Get disk info: [] E1121 10:59:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:59:06.411979 3798194 memory.go:184] no items to output this cycle I1121 10:59:06.412133 3798194 cpu.go:275] no items to output this cycle E1121 10:59:16.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:59:16.412874 3798194 memory.go:184] no items to output this cycle I1121 10:59:16.413017 3798194 cpu.go:275] no items to output this cycle E1121 10:59:26.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:59:26.413258 3798194 memory.go:191] Add success. I1121 10:59:26.417775 3798194 cpu.go:282] Add success. I1121 10:59:26.441440 3798194 net.go:642] Add success. I1121 10:59:26.442945 3798194 net.go:778] primary dev: ETH0 I1121 10:59:26.442960 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:59:26.442973 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:59:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:59:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:59:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 10:59:36.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:59:36.411985 3798194 memory.go:184] no items to output this cycle I1121 10:59:36.412139 3798194 cpu.go:275] no items to output this cycle E1121 10:59:46.411943 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:59:46.411965 3798194 memory.go:184] no items to output this cycle I1121 10:59:46.412108 3798194 cpu.go:275] no items to output this cycle E1121 10:59:56.413199 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 10:59:56.413234 3798194 memory.go:191] Add success. W1121 10:59:56.413263 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 10:59:56.413274 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 10:59:56.413278 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 10:59:56.417898 3798194 cpu.go:282] Add success. I1121 10:59:56.440960 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 10:59:56.441535 3798194 net.go:642] Add success. I1121 10:59:56.443090 3798194 net.go:778] primary dev: ETH0 I1121 10:59:56.443106 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 10:59:56.443136 3798194 net.go:659] Add eni metrics success, count: 7 I1121 10:59:56.591987 3798194 disk_info.go:125] begin check local disk info of client I1121 10:59:56.595975 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 10:59:56.595984 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071bc0 0xc000071c00] I1121 10:59:57.445897 3798194 custom_config.go:106] query custom config with name: gpu W1121 10:59:57.446169 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 10:59:57.446181 3798194 disk_worker.go:723] disk space is not compliant W1121 10:59:57.446184 3798194 disk_worker.go:743] disk inode is not compliant I1121 10:59:57.448326 3798194 disk_worker.go:509] system disk:vda1 I1121 10:59:57.448366 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 10:59:58.446374 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 10:59:59.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 10:59:59.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 10:59:59.450498 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 10:59:59.471058 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:00:06.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:00:06.412888 3798194 memory.go:184] no items to output this cycle I1121 11:00:06.413034 3798194 cpu.go:275] no items to output this cycle E1121 11:00:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:00:16.411978 3798194 memory.go:184] no items to output this cycle I1121 11:00:16.412141 3798194 cpu.go:275] no items to output this cycle I1121 11:00:17.444037 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:00:17.444047 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:00:26.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:00:26.412275 3798194 memory.go:191] Add success. I1121 11:00:26.416121 3798194 cpu.go:282] Add success. I1121 11:00:26.441742 3798194 net.go:642] Add success. I1121 11:00:26.443230 3798194 net.go:778] primary dev: ETH0 I1121 11:00:26.443245 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:00:26.443256 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:00:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:00:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:00:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:00:36.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:00:36.411974 3798194 memory.go:184] no items to output this cycle I1121 11:00:36.412127 3798194 cpu.go:275] no items to output this cycle E1121 11:00:46.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:00:46.413268 3798194 memory.go:184] no items to output this cycle I1121 11:00:46.413412 3798194 cpu.go:275] no items to output this cycle E1121 11:00:56.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:00:56.412274 3798194 memory.go:191] Add success. W1121 11:00:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:00:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:00:56.412316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:00:56.416540 3798194 cpu.go:282] Add success. I1121 11:00:56.445214 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:00:56.445689 3798194 net.go:642] Add success. I1121 11:00:56.447075 3798194 net.go:778] primary dev: ETH0 I1121 11:00:56.447090 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:00:56.447105 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:00:56.601149 3798194 disk_info.go:125] begin check local disk info of client I1121 11:00:56.612510 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:00:56.612520 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000407900 0xc000407940] E1121 11:00:57.450280 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 11:00:57.450294 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:00:57.450451 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:00:57.450501 3798194 disk_worker.go:723] disk space is not compliant W1121 11:00:57.450504 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:00:57.458909 3798194 disk_worker.go:509] system disk:vda1 I1121 11:00:57.458950 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 11:00:58.448152 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 11:00:58.448166 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 11:00:59.448801 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 11:00:59.449879 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:00:59.449936 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:00:59.449955 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:00:59.471532 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:01:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:01:06.412253 3798194 memory.go:184] no items to output this cycle I1121 11:01:06.412674 3798194 cpu.go:275] no items to output this cycle E1121 11:01:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:01:16.411978 3798194 memory.go:184] no items to output this cycle I1121 11:01:16.412145 3798194 cpu.go:275] no items to output this cycle E1121 11:01:26.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:01:26.412260 3798194 memory.go:191] Add success. I1121 11:01:26.416580 3798194 cpu.go:282] Add success. I1121 11:01:26.442096 3798194 net.go:642] Add success. I1121 11:01:26.443554 3798194 net.go:778] primary dev: ETH0 I1121 11:01:26.443569 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:01:26.443581 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:01:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:01:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:01:29.450151 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:01:36.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:01:36.411971 3798194 memory.go:184] no items to output this cycle I1121 11:01:36.412121 3798194 cpu.go:275] no items to output this cycle E1121 11:01:46.412839 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:01:46.412863 3798194 memory.go:184] no items to output this cycle I1121 11:01:46.413015 3798194 cpu.go:275] no items to output this cycle E1121 11:01:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:01:56.411949 3798194 memory.go:191] Add success. W1121 11:01:56.411974 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:01:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:01:56.411990 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:01:56.412330 3798194 cpu.go:282] Add success. I1121 11:01:56.441900 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:01:56.442491 3798194 net.go:642] Add success. I1121 11:01:56.446816 3798194 net.go:778] primary dev: ETH0 I1121 11:01:56.446831 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:01:56.446845 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:01:56.612628 3798194 disk_info.go:125] begin check local disk info of client I1121 11:01:56.616539 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:01:56.616548 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000351600 0xc000351640] I1121 11:01:57.450624 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:01:57.450822 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:01:57.450833 3798194 disk_worker.go:723] disk space is not compliant W1121 11:01:57.450836 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:01:57.459194 3798194 disk_worker.go:509] system disk:vda1 I1121 11:01:57.459236 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:01:58.446012 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:01:59.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:01:59.450487 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:01:59.450515 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:01:59.471038 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:02:06.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:02:06.413259 3798194 memory.go:184] no items to output this cycle I1121 11:02:06.413399 3798194 cpu.go:275] no items to output this cycle E1121 11:02:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:02:16.412256 3798194 memory.go:184] no items to output this cycle I1121 11:02:16.412402 3798194 cpu.go:275] no items to output this cycle E1121 11:02:26.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:02:26.412268 3798194 memory.go:191] Add success. I1121 11:02:26.416422 3798194 cpu.go:282] Add success. I1121 11:02:26.441965 3798194 net.go:642] Add success. I1121 11:02:26.443535 3798194 net.go:778] primary dev: ETH0 I1121 11:02:26.443551 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:02:26.443564 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:02:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:02:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:02:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:02:36.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:02:36.412258 3798194 memory.go:184] no items to output this cycle I1121 11:02:36.412402 3798194 cpu.go:275] no items to output this cycle E1121 11:02:46.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:02:46.412259 3798194 memory.go:184] no items to output this cycle I1121 11:02:46.412404 3798194 cpu.go:275] no items to output this cycle E1121 11:02:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:02:56.411948 3798194 memory.go:191] Add success. W1121 11:02:56.411974 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:02:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:02:56.411989 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:02:56.412352 3798194 cpu.go:282] Add success. I1121 11:02:56.441803 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:02:56.442304 3798194 net.go:642] Add success. I1121 11:02:56.446455 3798194 net.go:778] primary dev: ETH0 I1121 11:02:56.446469 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:02:56.446484 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:02:56.616650 3798194 disk_info.go:125] begin check local disk info of client I1121 11:02:56.620572 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:02:56.620581 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000351000 0xc000351040] I1121 11:02:57.450935 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:02:57.451104 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:02:57.451199 3798194 disk_worker.go:723] disk space is not compliant W1121 11:02:57.451203 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:02:57.459965 3798194 disk_worker.go:509] system disk:vda1 I1121 11:02:57.460024 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:02:58.446413 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:02:59.450411 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:02:59.450490 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:02:59.450515 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:02:59.471145 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:03:06.412760 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:03:06.412781 3798194 memory.go:184] no items to output this cycle I1121 11:03:06.412935 3798194 cpu.go:275] no items to output this cycle E1121 11:03:16.412018 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:03:16.412046 3798194 memory.go:184] no items to output this cycle I1121 11:03:16.412260 3798194 cpu.go:275] no items to output this cycle I1121 11:03:17.445068 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:03:17.445076 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:03:26.411289 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:03:26.411324 3798194 memory.go:191] Add success. I1121 11:03:26.411654 3798194 cpu.go:282] Add success. I1121 11:03:26.441251 3798194 net.go:642] Add success. I1121 11:03:26.444996 3798194 net.go:778] primary dev: ETH0 I1121 11:03:26.445012 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:03:26.445030 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:03:29.449673 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:03:29.449755 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:03:29.449781 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:03:36.412715 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:03:36.412738 3798194 memory.go:184] no items to output this cycle I1121 11:03:36.412884 3798194 cpu.go:275] no items to output this cycle E1121 11:03:46.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:03:46.413251 3798194 memory.go:184] no items to output this cycle I1121 11:03:46.413392 3798194 cpu.go:275] no items to output this cycle E1121 11:03:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:03:56.411941 3798194 memory.go:191] Add success. W1121 11:03:56.411966 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:03:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:03:56.411981 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:03:56.412306 3798194 cpu.go:282] Add success. I1121 11:03:56.441757 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:03:56.442349 3798194 net.go:642] Add success. I1121 11:03:56.446987 3798194 net.go:778] primary dev: ETH0 I1121 11:03:56.447002 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:03:56.447014 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:03:56.621602 3798194 disk_info.go:125] begin check local disk info of client I1121 11:03:56.625457 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:03:56.625465 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000471e80 0xc000471ec0] I1121 11:03:57.451274 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:03:57.451683 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:03:57.451696 3798194 disk_worker.go:723] disk space is not compliant W1121 11:03:57.451700 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:03:57.461063 3798194 disk_worker.go:509] system disk:vda1 I1121 11:03:57.461103 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:03:58.445700 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:03:59.449672 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:03:59.449747 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:03:59.449772 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:03:59.471185 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:04:06.412370 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:04:06.412392 3798194 memory.go:184] no items to output this cycle I1121 11:04:06.412539 3798194 cpu.go:275] no items to output this cycle E1121 11:04:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:04:16.412258 3798194 memory.go:184] no items to output this cycle I1121 11:04:16.412415 3798194 cpu.go:275] no items to output this cycle E1121 11:04:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:04:26.411953 3798194 memory.go:191] Add success. I1121 11:04:26.412315 3798194 cpu.go:282] Add success. I1121 11:04:26.441923 3798194 net.go:642] Add success. I1121 11:04:26.445999 3798194 net.go:778] primary dev: ETH0 I1121 11:04:26.446016 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:04:26.446030 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:04:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:04:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:04:29.450129 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:04:36.412084 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:04:36.412135 3798194 memory.go:184] no items to output this cycle I1121 11:04:36.412282 3798194 cpu.go:275] no items to output this cycle E1121 11:04:46.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:04:46.412263 3798194 memory.go:184] no items to output this cycle I1121 11:04:46.412412 3798194 cpu.go:275] no items to output this cycle E1121 11:04:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:04:56.412266 3798194 memory.go:191] Add success. W1121 11:04:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:04:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:04:56.412317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:04:56.416668 3798194 cpu.go:282] Add success. I1121 11:04:56.440956 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:04:56.441541 3798194 net.go:642] Add success. I1121 11:04:56.443003 3798194 net.go:778] primary dev: ETH0 I1121 11:04:56.443018 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:04:56.443031 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:04:56.626207 3798194 disk_info.go:125] begin check local disk info of client I1121 11:04:56.630147 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:04:56.630157 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b11c0 0xc0003b1200] I1121 11:04:57.450898 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:04:57.451107 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:04:57.451144 3798194 disk_worker.go:723] disk space is not compliant W1121 11:04:57.451147 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:04:57.459872 3798194 disk_worker.go:509] system disk:vda1 I1121 11:04:57.459916 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:04:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:04:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:04:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:04:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:04:59.471639 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:05:06.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:05:06.413260 3798194 memory.go:184] no items to output this cycle I1121 11:05:06.413414 3798194 cpu.go:275] no items to output this cycle E1121 11:05:16.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:05:16.412266 3798194 memory.go:184] no items to output this cycle I1121 11:05:16.412408 3798194 cpu.go:275] no items to output this cycle E1121 11:05:26.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:05:26.412274 3798194 memory.go:191] Add success. I1121 11:05:26.416645 3798194 cpu.go:282] Add success. I1121 11:05:26.442195 3798194 net.go:642] Add success. I1121 11:05:26.443701 3798194 net.go:778] primary dev: ETH0 I1121 11:05:26.443716 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:05:26.443728 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:05:29.450133 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:05:29.450214 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:05:29.450246 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:05:36.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:05:36.413255 3798194 memory.go:184] no items to output this cycle I1121 11:05:36.413394 3798194 cpu.go:275] no items to output this cycle E1121 11:05:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:05:46.413254 3798194 memory.go:184] no items to output this cycle I1121 11:05:46.413401 3798194 cpu.go:275] no items to output this cycle E1121 11:05:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:05:56.411949 3798194 memory.go:191] Add success. W1121 11:05:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:05:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:05:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:05:56.412342 3798194 cpu.go:282] Add success. I1121 11:05:56.441576 3798194 event_worker.go:152] Polling the log file for events... I1121 11:05:56.444844 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:05:56.445361 3798194 net.go:642] Add success. I1121 11:05:56.449195 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795228556, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 11:05:56.449319 3798194 net.go:778] primary dev: ETH0 I1121 11:05:56.449333 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:05:56.449354 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:05:56.457612 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 I1121 11:05:56.630257 3798194 disk_info.go:125] begin check local disk info of client I1121 11:05:56.634192 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:05:56.634202 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000447900 0xc000447940] E1121 11:05:57.445572 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 11:05:57.445589 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:05:57.445793 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:05:57.445856 3798194 disk_worker.go:723] disk space is not compliant W1121 11:05:57.445859 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:05:57.450812 3798194 disk_worker.go:509] system disk:vda1 I1121 11:05:57.450855 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 11:05:58.448152 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 11:05:58.448167 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 11:05:59.448982 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 11:05:59.450056 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:05:59.450117 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:05:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:05:59.471645 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:06:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:06:06.412259 3798194 memory.go:184] no items to output this cycle I1121 11:06:06.412403 3798194 cpu.go:275] no items to output this cycle E1121 11:06:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:06:16.411974 3798194 memory.go:184] no items to output this cycle I1121 11:06:16.412136 3798194 cpu.go:275] no items to output this cycle I1121 11:06:17.446013 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:06:17.446022 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:06:26.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:06:26.412261 3798194 memory.go:191] Add success. I1121 11:06:26.416584 3798194 cpu.go:282] Add success. I1121 11:06:26.442155 3798194 net.go:642] Add success. I1121 11:06:26.443698 3798194 net.go:778] primary dev: ETH0 I1121 11:06:26.443715 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:06:26.443729 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:06:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:06:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:06:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:06:36.413224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:06:36.413246 3798194 memory.go:184] no items to output this cycle I1121 11:06:36.413393 3798194 cpu.go:275] no items to output this cycle E1121 11:06:46.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:06:46.413263 3798194 memory.go:184] no items to output this cycle I1121 11:06:46.413616 3798194 cpu.go:275] no items to output this cycle E1121 11:06:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:06:56.411952 3798194 memory.go:191] Add success. W1121 11:06:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:06:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:06:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:06:56.412377 3798194 cpu.go:282] Add success. I1121 11:06:56.441900 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:06:56.442446 3798194 net.go:642] Add success. I1121 11:06:56.446431 3798194 net.go:778] primary dev: ETH0 I1121 11:06:56.446446 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:06:56.446459 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:06:56.634300 3798194 disk_info.go:125] begin check local disk info of client I1121 11:06:56.638341 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:06:56.638350 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e1040 0xc0002e1080] I1121 11:06:57.451071 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:06:57.451254 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:06:57.451311 3798194 disk_worker.go:723] disk space is not compliant W1121 11:06:57.451314 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:06:57.460164 3798194 disk_worker.go:509] system disk:vda1 I1121 11:06:57.460206 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:06:58.446009 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:06:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:06:59.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:06:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:06:59.471657 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:07:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:07:06.411982 3798194 memory.go:184] no items to output this cycle I1121 11:07:06.412142 3798194 cpu.go:275] no items to output this cycle E1121 11:07:16.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:07:16.412256 3798194 memory.go:184] no items to output this cycle I1121 11:07:16.412405 3798194 cpu.go:275] no items to output this cycle E1121 11:07:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:07:26.411948 3798194 memory.go:191] Add success. I1121 11:07:26.412298 3798194 cpu.go:282] Add success. I1121 11:07:26.442057 3798194 net.go:642] Add success. I1121 11:07:26.446378 3798194 net.go:778] primary dev: ETH0 I1121 11:07:26.446394 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:07:26.446406 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:07:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:07:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:07:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:07:36.412224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:07:36.412247 3798194 memory.go:184] no items to output this cycle I1121 11:07:36.412387 3798194 cpu.go:275] no items to output this cycle E1121 11:07:46.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:07:46.412885 3798194 memory.go:184] no items to output this cycle I1121 11:07:46.413028 3798194 cpu.go:275] no items to output this cycle E1121 11:07:56.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:07:56.412891 3798194 memory.go:191] Add success. W1121 11:07:56.412923 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:07:56.412935 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:07:56.412938 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:07:56.417277 3798194 cpu.go:282] Add success. I1121 11:07:56.441783 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:07:56.442289 3798194 net.go:642] Add success. I1121 11:07:56.443975 3798194 net.go:778] primary dev: ETH0 I1121 11:07:56.443992 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:07:56.444004 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:07:56.638442 3798194 disk_info.go:125] begin check local disk info of client I1121 11:07:56.642428 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:07:56.642437 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00045f840 0xc00045f880] I1121 11:07:57.450951 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:07:57.451194 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:07:57.451207 3798194 disk_worker.go:723] disk space is not compliant W1121 11:07:57.451210 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:07:57.460755 3798194 disk_worker.go:509] system disk:vda1 I1121 11:07:57.460793 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:07:58.446008 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:07:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:07:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:07:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:07:59.471642 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:08:06.412749 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:08:06.412771 3798194 memory.go:184] no items to output this cycle I1121 11:08:06.412914 3798194 cpu.go:275] no items to output this cycle E1121 11:08:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:08:16.411978 3798194 memory.go:184] no items to output this cycle I1121 11:08:16.412147 3798194 cpu.go:275] no items to output this cycle E1121 11:08:26.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:08:26.412274 3798194 memory.go:191] Add success. I1121 11:08:26.416539 3798194 cpu.go:282] Add success. I1121 11:08:26.442215 3798194 net.go:642] Add success. I1121 11:08:26.443713 3798194 net.go:778] primary dev: ETH0 I1121 11:08:26.443729 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:08:26.443744 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:08:29.450572 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:08:29.450644 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:08:29.450669 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:08:36.413264 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:08:36.413297 3798194 memory.go:184] no items to output this cycle I1121 11:08:36.413476 3798194 cpu.go:275] no items to output this cycle E1121 11:08:46.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:08:46.413263 3798194 memory.go:184] no items to output this cycle I1121 11:08:46.413413 3798194 cpu.go:275] no items to output this cycle E1121 11:08:56.412845 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:08:56.412881 3798194 memory.go:191] Add success. W1121 11:08:56.412915 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:08:56.412927 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:08:56.412930 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:08:56.417279 3798194 cpu.go:282] Add success. I1121 11:08:56.441786 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:08:56.442367 3798194 net.go:642] Add success. I1121 11:08:56.443845 3798194 net.go:778] primary dev: ETH0 I1121 11:08:56.443861 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:08:56.443873 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:08:56.642510 3798194 disk_info.go:125] begin check local disk info of client I1121 11:08:56.646572 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:08:56.646582 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003908c0 0xc000390900] I1121 11:08:57.450626 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:08:57.450834 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:08:57.450847 3798194 disk_worker.go:723] disk space is not compliant W1121 11:08:57.450850 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:08:57.459189 3798194 disk_worker.go:509] system disk:vda1 I1121 11:08:57.459232 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:08:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:08:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:08:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:08:59.450132 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:08:59.471699 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:09:06.411714 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:09:06.411734 3798194 memory.go:184] no items to output this cycle I1121 11:09:06.411877 3798194 cpu.go:275] no items to output this cycle E1121 11:09:16.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:09:16.411971 3798194 memory.go:184] no items to output this cycle I1121 11:09:16.412129 3798194 cpu.go:275] no items to output this cycle I1121 11:09:17.447080 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:09:17.447089 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:09:26.411352 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:09:26.411383 3798194 memory.go:191] Add success. I1121 11:09:26.411720 3798194 cpu.go:282] Add success. I1121 11:09:26.441309 3798194 net.go:642] Add success. I1121 11:09:26.445661 3798194 net.go:778] primary dev: ETH0 I1121 11:09:26.445676 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:09:26.445689 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:09:29.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:09:29.450463 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:09:29.450491 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:09:36.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:09:36.413276 3798194 memory.go:184] no items to output this cycle I1121 11:09:36.413427 3798194 cpu.go:275] no items to output this cycle E1121 11:09:46.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:09:46.413257 3798194 memory.go:184] no items to output this cycle I1121 11:09:46.413403 3798194 cpu.go:275] no items to output this cycle E1121 11:09:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:09:56.411948 3798194 memory.go:191] Add success. W1121 11:09:56.411981 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:09:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:09:56.411996 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:09:56.412358 3798194 cpu.go:282] Add success. I1121 11:09:56.441823 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:09:56.442420 3798194 net.go:642] Add success. I1121 11:09:56.446701 3798194 net.go:778] primary dev: ETH0 I1121 11:09:56.446716 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:09:56.446729 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:09:56.646690 3798194 disk_info.go:125] begin check local disk info of client I1121 11:09:56.650566 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:09:56.650575 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00027e0c0 0xc00027e100] I1121 11:09:57.451336 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:09:57.451497 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:09:57.451550 3798194 disk_worker.go:723] disk space is not compliant W1121 11:09:57.451553 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:09:57.459965 3798194 disk_worker.go:509] system disk:vda1 I1121 11:09:57.460006 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:09:58.445673 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:09:59.449693 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:09:59.449768 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:09:59.449793 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:09:59.471317 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:10:06.412964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:10:06.412984 3798194 memory.go:184] no items to output this cycle I1121 11:10:06.413143 3798194 cpu.go:275] no items to output this cycle E1121 11:10:16.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:10:16.413252 3798194 memory.go:184] no items to output this cycle I1121 11:10:16.413397 3798194 cpu.go:275] no items to output this cycle E1121 11:10:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:10:26.413265 3798194 memory.go:191] Add success. I1121 11:10:26.417524 3798194 cpu.go:282] Add success. I1121 11:10:26.442183 3798194 net.go:642] Add success. I1121 11:10:26.443649 3798194 net.go:778] primary dev: ETH0 I1121 11:10:26.443663 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:10:26.443680 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:10:29.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:10:29.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:10:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:10:36.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:10:36.413263 3798194 memory.go:184] no items to output this cycle I1121 11:10:36.413406 3798194 cpu.go:275] no items to output this cycle E1121 11:10:46.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:10:46.412265 3798194 memory.go:184] no items to output this cycle I1121 11:10:46.412410 3798194 cpu.go:275] no items to output this cycle E1121 11:10:56.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:10:56.411947 3798194 memory.go:191] Add success. W1121 11:10:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:10:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:10:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:10:56.412343 3798194 cpu.go:282] Add success. I1121 11:10:56.443823 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:10:56.444322 3798194 net.go:642] Add success. I1121 11:10:56.448344 3798194 net.go:778] primary dev: ETH0 I1121 11:10:56.448359 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:10:56.448371 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:10:56.650677 3798194 disk_info.go:125] begin check local disk info of client I1121 11:10:56.654485 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:10:56.654505 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071280 0xc0000712c0] E1121 11:10:57.450591 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 11:10:57.450606 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:10:57.450806 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:10:57.450816 3798194 disk_worker.go:723] disk space is not compliant W1121 11:10:57.450819 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:10:57.459159 3798194 disk_worker.go:509] system disk:vda1 I1121 11:10:57.459203 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 11:10:58.448057 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 11:10:58.448072 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 11:10:59.448612 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 11:10:59.449687 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:10:59.449737 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:10:59.449755 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:10:59.471250 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:11:06.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:11:06.411987 3798194 memory.go:184] no items to output this cycle I1121 11:11:06.412142 3798194 cpu.go:275] no items to output this cycle E1121 11:11:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:11:16.411970 3798194 memory.go:184] no items to output this cycle I1121 11:11:16.412136 3798194 cpu.go:275] no items to output this cycle E1121 11:11:26.413673 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:11:26.413707 3798194 memory.go:191] Add success. I1121 11:11:26.418007 3798194 cpu.go:282] Add success. I1121 11:11:26.441664 3798194 net.go:642] Add success. I1121 11:11:26.443019 3798194 net.go:778] primary dev: ETH0 I1121 11:11:26.443032 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:11:26.443045 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:11:29.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:11:29.450490 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:11:29.450514 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:11:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:11:36.411980 3798194 memory.go:184] no items to output this cycle I1121 11:11:36.412140 3798194 cpu.go:275] no items to output this cycle E1121 11:11:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:11:46.411978 3798194 memory.go:184] no items to output this cycle I1121 11:11:46.412132 3798194 cpu.go:275] no items to output this cycle E1121 11:11:56.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:11:56.413268 3798194 memory.go:191] Add success. W1121 11:11:56.413296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:11:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:11:56.413313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:11:56.417728 3798194 cpu.go:282] Add success. I1121 11:11:56.441076 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:11:56.441565 3798194 net.go:642] Add success. I1121 11:11:56.443040 3798194 net.go:778] primary dev: ETH0 I1121 11:11:56.443055 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:11:56.443068 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:11:56.654598 3798194 disk_info.go:125] begin check local disk info of client I1121 11:11:56.658436 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:11:56.658445 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000417600 0xc000417640] I1121 11:11:57.450928 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:11:57.451084 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:11:57.451166 3798194 disk_worker.go:723] disk space is not compliant W1121 11:11:57.451169 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:11:57.460010 3798194 disk_worker.go:509] system disk:vda1 I1121 11:11:57.460053 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:11:58.446386 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:11:59.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:11:59.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:11:59.450147 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:11:59.471692 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:12:06.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:12:06.412877 3798194 memory.go:184] no items to output this cycle I1121 11:12:06.413243 3798194 cpu.go:275] no items to output this cycle E1121 11:12:16.411790 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:12:16.411812 3798194 memory.go:184] no items to output this cycle I1121 11:12:16.411957 3798194 cpu.go:275] no items to output this cycle I1121 11:12:17.447449 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:12:17.447457 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:12:26.416237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:12:26.416275 3798194 memory.go:191] Add success. I1121 11:12:26.437341 3798194 cpu.go:282] Add success. I1121 11:12:26.445864 3798194 net.go:642] Add success. I1121 11:12:26.466274 3798194 net.go:778] primary dev: ETH0 I1121 11:12:26.466292 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:12:26.466305 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:12:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:12:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:12:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:12:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:12:36.411979 3798194 memory.go:184] no items to output this cycle I1121 11:12:36.412134 3798194 cpu.go:275] no items to output this cycle E1121 11:12:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:12:46.413262 3798194 memory.go:184] no items to output this cycle I1121 11:12:46.413409 3798194 cpu.go:275] no items to output this cycle E1121 11:12:56.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:12:56.413265 3798194 memory.go:191] Add success. W1121 11:12:56.413292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:12:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:12:56.413307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:12:56.418040 3798194 cpu.go:282] Add success. I1121 11:12:56.441505 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:12:56.442047 3798194 net.go:642] Add success. I1121 11:12:56.443573 3798194 net.go:778] primary dev: ETH0 I1121 11:12:56.443588 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:12:56.443601 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:12:56.658548 3798194 disk_info.go:125] begin check local disk info of client I1121 11:12:56.662454 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:12:56.662463 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00034b600 0xc00034b640] I1121 11:12:57.445515 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:12:57.445725 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:12:57.445790 3798194 disk_worker.go:723] disk space is not compliant W1121 11:12:57.445793 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:12:57.447892 3798194 disk_worker.go:509] system disk:vda1 I1121 11:12:57.447934 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:12:58.446442 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:12:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:12:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:12:59.450148 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:12:59.471689 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:13:06.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:13:06.412883 3798194 memory.go:184] no items to output this cycle I1121 11:13:06.413021 3798194 cpu.go:275] no items to output this cycle E1121 11:13:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:13:16.411980 3798194 memory.go:184] no items to output this cycle I1121 11:13:16.412143 3798194 cpu.go:275] no items to output this cycle E1121 11:13:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:13:26.413267 3798194 memory.go:191] Add success. I1121 11:13:26.417868 3798194 cpu.go:282] Add success. I1121 11:13:26.441576 3798194 net.go:642] Add success. I1121 11:13:26.443221 3798194 net.go:778] primary dev: ETH0 I1121 11:13:26.443239 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:13:26.443251 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:13:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:13:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:13:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:13:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:13:36.413260 3798194 memory.go:184] no items to output this cycle I1121 11:13:36.413403 3798194 cpu.go:275] no items to output this cycle E1121 11:13:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:13:46.411991 3798194 memory.go:184] no items to output this cycle I1121 11:13:46.412165 3798194 cpu.go:275] no items to output this cycle E1121 11:13:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:13:56.411954 3798194 memory.go:191] Add success. W1121 11:13:56.411981 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:13:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:13:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:13:56.412374 3798194 cpu.go:282] Add success. I1121 11:13:56.441823 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:13:56.442341 3798194 net.go:642] Add success. I1121 11:13:56.446383 3798194 net.go:778] primary dev: ETH0 I1121 11:13:56.446398 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:13:56.446411 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:13:56.662562 3798194 disk_info.go:125] begin check local disk info of client I1121 11:13:56.666378 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:13:56.666386 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000371780 0xc0003717c0] I1121 11:13:57.450579 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:13:57.450777 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:13:57.450788 3798194 disk_worker.go:723] disk space is not compliant W1121 11:13:57.450791 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:13:57.459176 3798194 disk_worker.go:509] system disk:vda1 I1121 11:13:57.459222 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:13:58.446010 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:13:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:13:59.450124 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:13:59.450152 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:13:59.471671 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:14:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:14:06.412253 3798194 memory.go:184] no items to output this cycle I1121 11:14:06.412402 3798194 cpu.go:275] no items to output this cycle E1121 11:14:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:14:16.411975 3798194 memory.go:184] no items to output this cycle I1121 11:14:16.412133 3798194 cpu.go:275] no items to output this cycle E1121 11:14:26.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:14:26.413277 3798194 memory.go:191] Add success. I1121 11:14:26.417206 3798194 cpu.go:282] Add success. I1121 11:14:26.441733 3798194 net.go:642] Add success. I1121 11:14:26.443138 3798194 net.go:778] primary dev: ETH0 I1121 11:14:26.443153 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:14:26.443168 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:14:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:14:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:14:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:14:36.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:14:36.412263 3798194 memory.go:184] no items to output this cycle I1121 11:14:36.412409 3798194 cpu.go:275] no items to output this cycle E1121 11:14:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:14:46.413263 3798194 memory.go:184] no items to output this cycle I1121 11:14:46.413410 3798194 cpu.go:275] no items to output this cycle E1121 11:14:56.411909 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:14:56.411943 3798194 memory.go:191] Add success. W1121 11:14:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:14:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:14:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:14:56.412337 3798194 cpu.go:282] Add success. I1121 11:14:56.441790 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:14:56.442295 3798194 net.go:642] Add success. I1121 11:14:56.446710 3798194 net.go:778] primary dev: ETH0 I1121 11:14:56.446725 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:14:56.446736 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:14:56.666482 3798194 disk_info.go:125] begin check local disk info of client I1121 11:14:56.670410 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:14:56.670418 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047fe40 0xc00047fe80] I1121 11:14:57.450593 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:14:57.450764 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:14:57.450820 3798194 disk_worker.go:723] disk space is not compliant W1121 11:14:57.450824 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:14:57.459357 3798194 disk_worker.go:509] system disk:vda1 I1121 11:14:57.459412 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:14:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:14:59.450186 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:14:59.450267 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:14:59.450293 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:14:59.471670 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:15:06.412536 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:15:06.412558 3798194 memory.go:184] no items to output this cycle I1121 11:15:06.412703 3798194 cpu.go:275] no items to output this cycle E1121 11:15:16.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:15:16.412256 3798194 memory.go:184] no items to output this cycle I1121 11:15:16.412408 3798194 cpu.go:275] no items to output this cycle I1121 11:15:17.447605 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:15:17.447613 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:15:26.411344 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:15:26.411375 3798194 memory.go:191] Add success. I1121 11:15:26.411695 3798194 cpu.go:282] Add success. I1121 11:15:26.441455 3798194 net.go:642] Add success. I1121 11:15:26.445243 3798194 net.go:778] primary dev: ETH0 I1121 11:15:26.445259 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:15:26.445272 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:15:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:15:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:15:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:15:36.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:15:36.411987 3798194 memory.go:184] no items to output this cycle I1121 11:15:36.412141 3798194 cpu.go:275] no items to output this cycle E1121 11:15:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:15:46.413258 3798194 memory.go:184] no items to output this cycle I1121 11:15:46.413404 3798194 cpu.go:275] no items to output this cycle E1121 11:15:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:15:56.411954 3798194 memory.go:191] Add success. W1121 11:15:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:15:56.412005 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:15:56.412010 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:15:56.412361 3798194 cpu.go:282] Add success. I1121 11:15:56.441570 3798194 event_worker.go:152] Polling the log file for events... I1121 11:15:56.444726 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:15:56.445265 3798194 net.go:642] Add success. I1121 11:15:56.449185 3798194 net.go:778] primary dev: ETH0 I1121 11:15:56.449202 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:15:56.449214 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:15:56.670520 3798194 disk_info.go:125] begin check local disk info of client I1121 11:15:56.674435 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:15:56.674444 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00039e880 0xc00039e8c0] E1121 11:15:57.451005 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 11:15:57.451019 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:15:57.451251 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:15:57.451261 3798194 disk_worker.go:723] disk space is not compliant W1121 11:15:57.451264 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:15:57.459731 3798194 disk_worker.go:509] system disk:vda1 I1121 11:15:57.459773 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 11:15:58.448648 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 11:15:58.448664 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 11:15:59.448621 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 11:15:59.449698 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:15:59.449752 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:15:59.449772 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:15:59.471303 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:16:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:16:06.411978 3798194 memory.go:184] no items to output this cycle I1121 11:16:06.412138 3798194 cpu.go:275] no items to output this cycle E1121 11:16:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:16:16.412256 3798194 memory.go:184] no items to output this cycle I1121 11:16:16.412405 3798194 cpu.go:275] no items to output this cycle E1121 11:16:26.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:16:26.412276 3798194 memory.go:191] Add success. I1121 11:16:26.416454 3798194 cpu.go:282] Add success. I1121 11:16:26.442105 3798194 net.go:642] Add success. I1121 11:16:26.443620 3798194 net.go:778] primary dev: ETH0 I1121 11:16:26.443636 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:16:26.443665 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:16:29.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:16:29.450125 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:16:29.450155 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:16:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:16:36.413257 3798194 memory.go:184] no items to output this cycle I1121 11:16:36.413401 3798194 cpu.go:275] no items to output this cycle E1121 11:16:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:16:46.413264 3798194 memory.go:184] no items to output this cycle I1121 11:16:46.413405 3798194 cpu.go:275] no items to output this cycle E1121 11:16:56.411312 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:16:56.411346 3798194 memory.go:191] Add success. W1121 11:16:56.411386 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:16:56.411397 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:16:56.411400 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:16:56.415615 3798194 cpu.go:282] Add success. I1121 11:16:56.440955 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:16:56.441554 3798194 net.go:642] Add success. I1121 11:16:56.443020 3798194 net.go:778] primary dev: ETH0 I1121 11:16:56.443035 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:16:56.443047 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:16:56.674545 3798194 disk_info.go:125] begin check local disk info of client I1121 11:16:56.678855 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:16:56.678865 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003dad40 0xc0003dad80] I1121 11:16:57.450551 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:16:57.450760 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:16:57.450772 3798194 disk_worker.go:723] disk space is not compliant W1121 11:16:57.450775 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:16:57.459137 3798194 disk_worker.go:509] system disk:vda1 I1121 11:16:57.459178 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:16:58.446375 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:16:59.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:16:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:16:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:16:59.471721 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:17:06.412865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:17:06.412885 3798194 memory.go:184] no items to output this cycle I1121 11:17:06.413029 3798194 cpu.go:275] no items to output this cycle E1121 11:17:16.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:17:16.412883 3798194 memory.go:184] no items to output this cycle I1121 11:17:16.413034 3798194 cpu.go:275] no items to output this cycle E1121 11:17:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:17:26.411956 3798194 memory.go:191] Add success. I1121 11:17:26.412291 3798194 cpu.go:282] Add success. I1121 11:17:26.441896 3798194 net.go:642] Add success. I1121 11:17:26.446103 3798194 net.go:778] primary dev: ETH0 I1121 11:17:26.446132 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:17:26.446146 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:17:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:17:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:17:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:17:36.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:17:36.412259 3798194 memory.go:184] no items to output this cycle I1121 11:17:36.412396 3798194 cpu.go:275] no items to output this cycle E1121 11:17:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:17:46.411978 3798194 memory.go:184] no items to output this cycle I1121 11:17:46.412133 3798194 cpu.go:275] no items to output this cycle E1121 11:17:56.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:17:56.412897 3798194 memory.go:191] Add success. W1121 11:17:56.412932 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:17:56.412944 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:17:56.412948 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:17:56.417226 3798194 cpu.go:282] Add success. I1121 11:17:56.441779 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:17:56.442568 3798194 net.go:642] Add success. I1121 11:17:56.444123 3798194 net.go:778] primary dev: ETH0 I1121 11:17:56.444139 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:17:56.444152 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:17:56.678973 3798194 disk_info.go:125] begin check local disk info of client I1121 11:17:56.683008 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:17:56.683017 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004e2080 0xc0004e20c0] I1121 11:17:57.450607 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:17:57.450768 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:17:57.450838 3798194 disk_worker.go:723] disk space is not compliant W1121 11:17:57.450842 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:17:57.459239 3798194 disk_worker.go:509] system disk:vda1 I1121 11:17:57.459278 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:17:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:17:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:17:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:17:59.450132 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:17:59.471641 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:18:06.412847 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:18:06.412869 3798194 memory.go:184] no items to output this cycle I1121 11:18:06.413011 3798194 cpu.go:275] no items to output this cycle E1121 11:18:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:18:16.411982 3798194 memory.go:184] no items to output this cycle I1121 11:18:16.412144 3798194 cpu.go:275] no items to output this cycle I1121 11:18:17.448097 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:18:17.448106 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:18:26.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:18:26.412894 3798194 memory.go:191] Add success. I1121 11:18:26.417297 3798194 cpu.go:282] Add success. I1121 11:18:26.441976 3798194 net.go:642] Add success. I1121 11:18:26.443600 3798194 net.go:778] primary dev: ETH0 I1121 11:18:26.443616 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:18:26.443629 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:18:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:18:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:18:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:18:36.412290 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:18:36.412319 3798194 memory.go:184] no items to output this cycle I1121 11:18:36.412527 3798194 cpu.go:275] no items to output this cycle E1121 11:18:46.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:18:46.412275 3798194 memory.go:184] no items to output this cycle I1121 11:18:46.412426 3798194 cpu.go:275] no items to output this cycle E1121 11:18:56.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:18:56.413270 3798194 memory.go:191] Add success. W1121 11:18:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:18:56.413318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:18:56.413321 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:18:56.418165 3798194 cpu.go:282] Add success. I1121 11:18:56.441581 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:18:56.442186 3798194 net.go:642] Add success. I1121 11:18:56.443661 3798194 net.go:778] primary dev: ETH0 I1121 11:18:56.443676 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:18:56.443689 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:18:56.683120 3798194 disk_info.go:125] begin check local disk info of client I1121 11:18:56.688167 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:18:56.688176 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c6ec0 0xc0003c6f00] I1121 11:18:57.445904 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:18:57.446181 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:18:57.446193 3798194 disk_worker.go:723] disk space is not compliant W1121 11:18:57.446196 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:18:57.448304 3798194 disk_worker.go:509] system disk:vda1 I1121 11:18:57.448343 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:18:58.446386 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:18:59.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:18:59.450490 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:18:59.450516 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:18:59.471054 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:19:06.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:19:06.413248 3798194 memory.go:184] no items to output this cycle I1121 11:19:06.413397 3798194 cpu.go:275] no items to output this cycle E1121 11:19:16.412725 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:19:16.412747 3798194 memory.go:184] no items to output this cycle I1121 11:19:16.412907 3798194 cpu.go:275] no items to output this cycle E1121 11:19:26.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:19:26.412283 3798194 memory.go:191] Add success. I1121 11:19:26.417165 3798194 cpu.go:282] Add success. I1121 11:19:26.441762 3798194 net.go:642] Add success. I1121 11:19:26.443338 3798194 net.go:778] primary dev: ETH0 I1121 11:19:26.443353 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:19:26.443366 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:19:29.449678 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:19:29.449748 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:19:29.449781 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:19:36.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:19:36.412249 3798194 memory.go:184] no items to output this cycle I1121 11:19:36.412399 3798194 cpu.go:275] no items to output this cycle E1121 11:19:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:19:46.411978 3798194 memory.go:184] no items to output this cycle I1121 11:19:46.412143 3798194 cpu.go:275] no items to output this cycle E1121 11:19:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:19:56.411944 3798194 memory.go:191] Add success. W1121 11:19:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:19:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:19:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:19:56.412329 3798194 cpu.go:282] Add success. I1121 11:19:56.441929 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:19:56.442512 3798194 net.go:642] Add success. I1121 11:19:56.455572 3798194 net.go:778] primary dev: ETH0 I1121 11:19:56.455588 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:19:56.455600 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:19:56.688277 3798194 disk_info.go:125] begin check local disk info of client I1121 11:19:56.692218 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:19:56.692227 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002f9a80 0xc0002f9ac0] I1121 11:19:57.450625 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:19:57.450834 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:19:57.450845 3798194 disk_worker.go:723] disk space is not compliant W1121 11:19:57.450848 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:19:57.459228 3798194 disk_worker.go:509] system disk:vda1 I1121 11:19:57.459267 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:19:58.445997 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:19:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:19:59.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:19:59.450498 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:19:59.471023 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:20:06.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:20:06.413261 3798194 memory.go:184] no items to output this cycle I1121 11:20:06.413404 3798194 cpu.go:275] no items to output this cycle E1121 11:20:16.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:20:16.411986 3798194 memory.go:184] no items to output this cycle I1121 11:20:16.412142 3798194 cpu.go:275] no items to output this cycle E1121 11:20:26.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:20:26.413272 3798194 memory.go:191] Add success. I1121 11:20:26.417660 3798194 cpu.go:282] Add success. I1121 11:20:26.441402 3798194 net.go:642] Add success. I1121 11:20:26.442861 3798194 net.go:778] primary dev: ETH0 I1121 11:20:26.442876 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:20:26.442889 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:20:29.449886 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:20:29.449970 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:20:29.449998 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:20:36.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:20:36.413264 3798194 memory.go:184] no items to output this cycle I1121 11:20:36.413410 3798194 cpu.go:275] no items to output this cycle E1121 11:20:46.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:20:46.411985 3798194 memory.go:184] no items to output this cycle I1121 11:20:46.412142 3798194 cpu.go:275] no items to output this cycle E1121 11:20:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:20:56.411956 3798194 memory.go:191] Add success. W1121 11:20:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:20:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:20:56.412002 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:20:56.412880 3798194 cpu.go:282] Add success. I1121 11:20:56.444726 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:20:56.445255 3798194 net.go:642] Add success. I1121 11:20:56.449353 3798194 net.go:778] primary dev: ETH0 I1121 11:20:56.449368 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:20:56.449381 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:20:56.693168 3798194 disk_info.go:125] begin check local disk info of client I1121 11:20:56.697004 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:20:56.697013 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071100 0xc000071140] E1121 11:20:57.450576 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 11:20:57.450592 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:20:57.450743 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:20:57.450810 3798194 disk_worker.go:723] disk space is not compliant W1121 11:20:57.450814 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:20:57.459217 3798194 disk_worker.go:509] system disk:vda1 I1121 11:20:57.459264 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 11:20:58.448064 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 11:20:58.448079 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 11:20:59.449595 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 11:20:59.449611 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:20:59.449670 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:20:59.449692 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:20:59.471185 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:21:06.413088 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:21:06.413123 3798194 memory.go:184] no items to output this cycle I1121 11:21:06.413276 3798194 cpu.go:275] no items to output this cycle E1121 11:21:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:21:16.411975 3798194 memory.go:184] no items to output this cycle I1121 11:21:16.412134 3798194 cpu.go:275] no items to output this cycle I1121 11:21:17.449085 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:21:17.449094 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:21:26.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:21:26.413269 3798194 memory.go:191] Add success. I1121 11:21:26.417628 3798194 cpu.go:282] Add success. I1121 11:21:26.442165 3798194 net.go:642] Add success. I1121 11:21:26.443786 3798194 net.go:778] primary dev: ETH0 I1121 11:21:26.443801 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:21:26.443813 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:21:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:21:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:21:29.450495 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:21:36.411936 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:21:36.411959 3798194 memory.go:184] no items to output this cycle I1121 11:21:36.412100 3798194 cpu.go:275] no items to output this cycle E1121 11:21:46.413404 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:21:46.413426 3798194 memory.go:184] no items to output this cycle I1121 11:21:46.413571 3798194 cpu.go:275] no items to output this cycle E1121 11:21:56.412719 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:21:56.412753 3798194 memory.go:191] Add success. W1121 11:21:56.412787 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:21:56.412799 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:21:56.412804 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:21:56.417359 3798194 cpu.go:282] Add success. I1121 11:21:56.441751 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:21:56.442272 3798194 net.go:642] Add success. I1121 11:21:56.443723 3798194 net.go:778] primary dev: ETH0 I1121 11:21:56.443736 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:21:56.443750 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:21:56.697141 3798194 disk_info.go:125] begin check local disk info of client I1121 11:21:56.700924 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:21:56.700933 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040c4c0 0xc00040c500] I1121 11:21:57.450869 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:21:57.451071 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:21:57.451082 3798194 disk_worker.go:723] disk space is not compliant W1121 11:21:57.451085 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:21:57.459834 3798194 disk_worker.go:509] system disk:vda1 I1121 11:21:57.459877 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:21:58.446141 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:21:59.450191 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:21:59.450281 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:21:59.450308 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:21:59.470844 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:22:06.412876 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:22:06.412898 3798194 memory.go:184] no items to output this cycle I1121 11:22:06.413047 3798194 cpu.go:275] no items to output this cycle E1121 11:22:16.412782 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:22:16.412803 3798194 memory.go:184] no items to output this cycle I1121 11:22:16.412963 3798194 cpu.go:275] no items to output this cycle E1121 11:22:26.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:22:26.412265 3798194 memory.go:191] Add success. I1121 11:22:26.416642 3798194 cpu.go:282] Add success. I1121 11:22:26.442169 3798194 net.go:642] Add success. I1121 11:22:26.443700 3798194 net.go:778] primary dev: ETH0 I1121 11:22:26.443714 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:22:26.443728 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:22:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:22:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:22:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:22:36.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:22:36.412888 3798194 memory.go:184] no items to output this cycle I1121 11:22:36.413030 3798194 cpu.go:275] no items to output this cycle E1121 11:22:46.412839 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:22:46.412862 3798194 memory.go:184] no items to output this cycle I1121 11:22:46.413011 3798194 cpu.go:275] no items to output this cycle E1121 11:22:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:22:56.412267 3798194 memory.go:191] Add success. W1121 11:22:56.412301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:22:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:22:56.412316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:22:56.416784 3798194 cpu.go:282] Add success. I1121 11:22:56.441174 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:22:56.441648 3798194 net.go:642] Add success. I1121 11:22:56.443234 3798194 net.go:778] primary dev: ETH0 I1121 11:22:56.443251 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:22:56.443264 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:22:56.702146 3798194 disk_info.go:125] begin check local disk info of client I1121 11:22:56.706147 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:22:56.706156 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000398700 0xc000398740] I1121 11:22:57.450503 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:22:57.450677 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:22:57.450737 3798194 disk_worker.go:723] disk space is not compliant W1121 11:22:57.450740 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:22:57.459161 3798194 disk_worker.go:509] system disk:vda1 I1121 11:22:57.459203 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:22:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:22:59.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:22:59.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:22:59.450501 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:22:59.471039 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:23:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:23:06.411975 3798194 memory.go:184] no items to output this cycle I1121 11:23:06.412144 3798194 cpu.go:275] no items to output this cycle E1121 11:23:16.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:23:16.412260 3798194 memory.go:184] no items to output this cycle I1121 11:23:16.412403 3798194 cpu.go:275] no items to output this cycle E1121 11:23:26.412268 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:23:26.412301 3798194 memory.go:191] Add success. I1121 11:23:26.416411 3798194 cpu.go:282] Add success. I1121 11:23:26.442333 3798194 net.go:642] Add success. I1121 11:23:26.443825 3798194 net.go:778] primary dev: ETH0 I1121 11:23:26.443842 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:23:26.443861 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:23:29.449672 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:23:29.449762 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:23:29.449790 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:23:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:23:36.411979 3798194 memory.go:184] no items to output this cycle I1121 11:23:36.412129 3798194 cpu.go:275] no items to output this cycle E1121 11:23:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:23:46.411983 3798194 memory.go:184] no items to output this cycle I1121 11:23:46.412142 3798194 cpu.go:275] no items to output this cycle E1121 11:23:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:23:56.413268 3798194 memory.go:191] Add success. W1121 11:23:56.413299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:23:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:23:56.413314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:23:56.417789 3798194 cpu.go:282] Add success. I1121 11:23:56.441192 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:23:56.441666 3798194 net.go:642] Add success. I1121 11:23:56.443245 3798194 net.go:778] primary dev: ETH0 I1121 11:23:56.443260 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:23:56.443273 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:23:56.707147 3798194 disk_info.go:125] begin check local disk info of client I1121 11:23:56.711067 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:23:56.711076 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a6180 0xc0004a61c0] I1121 11:23:57.450394 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:23:57.450556 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:23:57.450611 3798194 disk_worker.go:723] disk space is not compliant W1121 11:23:57.450614 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:23:57.459006 3798194 disk_worker.go:509] system disk:vda1 I1121 11:23:57.459048 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:23:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:23:59.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:23:59.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:23:59.450499 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:23:59.471016 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:24:06.411411 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:24:06.411433 3798194 memory.go:184] no items to output this cycle I1121 11:24:06.411589 3798194 cpu.go:275] no items to output this cycle E1121 11:24:16.412818 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:24:16.412842 3798194 memory.go:184] no items to output this cycle I1121 11:24:16.413004 3798194 cpu.go:275] no items to output this cycle I1121 11:24:17.450143 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:24:17.450151 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:24:26.411363 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:24:26.411397 3798194 memory.go:191] Add success. I1121 11:24:26.411760 3798194 cpu.go:282] Add success. I1121 11:24:26.441306 3798194 net.go:642] Add success. I1121 11:24:26.445542 3798194 net.go:778] primary dev: ETH0 I1121 11:24:26.445558 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:24:26.445570 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:24:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:24:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:24:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:24:36.412819 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:24:36.412842 3798194 memory.go:184] no items to output this cycle I1121 11:24:36.413010 3798194 cpu.go:275] no items to output this cycle E1121 11:24:46.413637 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:24:46.413660 3798194 memory.go:184] no items to output this cycle I1121 11:24:46.413804 3798194 cpu.go:275] no items to output this cycle E1121 11:24:56.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:24:56.413276 3798194 memory.go:191] Add success. W1121 11:24:56.413310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:24:56.413321 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:24:56.413324 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:24:56.417998 3798194 cpu.go:282] Add success. I1121 11:24:56.441250 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:24:56.441735 3798194 net.go:642] Add success. I1121 11:24:56.443108 3798194 net.go:778] primary dev: ETH0 I1121 11:24:56.443140 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:24:56.443154 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:24:56.711691 3798194 disk_info.go:125] begin check local disk info of client I1121 11:24:56.715829 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:24:56.715838 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071280 0xc0000712c0] I1121 11:24:57.445561 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:24:57.445777 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:24:57.445846 3798194 disk_worker.go:723] disk space is not compliant W1121 11:24:57.445849 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:24:57.450790 3798194 disk_worker.go:509] system disk:vda1 I1121 11:24:57.450830 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:24:58.445988 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:24:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:24:59.450119 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:24:59.450151 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:24:59.471647 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:25:06.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:25:06.412258 3798194 memory.go:184] no items to output this cycle I1121 11:25:06.412402 3798194 cpu.go:275] no items to output this cycle E1121 11:25:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:25:16.411979 3798194 memory.go:184] no items to output this cycle I1121 11:25:16.412136 3798194 cpu.go:275] no items to output this cycle E1121 11:25:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:25:26.411951 3798194 memory.go:191] Add success. I1121 11:25:26.412311 3798194 cpu.go:282] Add success. I1121 11:25:26.441917 3798194 net.go:642] Add success. I1121 11:25:26.446077 3798194 net.go:778] primary dev: ETH0 I1121 11:25:26.446092 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:25:26.446104 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:25:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:25:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:25:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:25:36.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:25:36.412267 3798194 memory.go:184] no items to output this cycle I1121 11:25:36.412414 3798194 cpu.go:275] no items to output this cycle E1121 11:25:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:25:46.411978 3798194 memory.go:184] no items to output this cycle I1121 11:25:46.412141 3798194 cpu.go:275] no items to output this cycle E1121 11:25:56.411909 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:25:56.411940 3798194 memory.go:191] Add success. W1121 11:25:56.411974 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:25:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:25:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:25:56.412336 3798194 cpu.go:282] Add success. I1121 11:25:56.441554 3798194 event_worker.go:152] Polling the log file for events... I1121 11:25:56.444728 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:25:56.445259 3798194 net.go:642] Add success. I1121 11:25:56.449254 3798194 net.go:778] primary dev: ETH0 I1121 11:25:56.449269 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:25:56.449283 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:25:56.716135 3798194 disk_info.go:125] begin check local disk info of client I1121 11:25:56.720093 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:25:56.720101 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000432380 0xc0004323c0] E1121 11:25:57.450955 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 11:25:57.450971 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:25:57.451151 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:25:57.451207 3798194 disk_worker.go:723] disk space is not compliant W1121 11:25:57.451210 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:25:57.459846 3798194 disk_worker.go:509] system disk:vda1 I1121 11:25:57.459888 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 11:25:58.448137 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 11:25:58.448152 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 11:25:59.448610 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 11:25:59.449683 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:25:59.449734 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:25:59.449754 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:25:59.471314 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:26:06.412769 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:26:06.412792 3798194 memory.go:184] no items to output this cycle I1121 11:26:06.412942 3798194 cpu.go:275] no items to output this cycle E1121 11:26:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:26:16.412255 3798194 memory.go:184] no items to output this cycle I1121 11:26:16.412399 3798194 cpu.go:275] no items to output this cycle E1121 11:26:26.412840 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:26:26.412876 3798194 memory.go:191] Add success. I1121 11:26:26.417333 3798194 cpu.go:282] Add success. I1121 11:26:26.441991 3798194 net.go:642] Add success. I1121 11:26:26.443667 3798194 net.go:778] primary dev: ETH0 I1121 11:26:26.443681 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:26:26.443695 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:26:29.450033 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:26:29.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:26:29.450150 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:26:36.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:26:36.411973 3798194 memory.go:184] no items to output this cycle I1121 11:26:36.412132 3798194 cpu.go:275] no items to output this cycle E1121 11:26:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:26:46.413255 3798194 memory.go:184] no items to output this cycle I1121 11:26:46.413406 3798194 cpu.go:275] no items to output this cycle E1121 11:26:56.413197 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:26:56.413231 3798194 memory.go:191] Add success. W1121 11:26:56.413264 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:26:56.413275 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:26:56.413279 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:26:56.417531 3798194 cpu.go:282] Add success. I1121 11:26:56.440917 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:26:56.441577 3798194 net.go:642] Add success. I1121 11:26:56.443168 3798194 net.go:778] primary dev: ETH0 I1121 11:26:56.443184 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:26:56.443197 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:26:56.721156 3798194 disk_info.go:125] begin check local disk info of client I1121 11:26:56.725174 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:26:56.725185 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000351dc0 0xc000351e00] I1121 11:26:57.450647 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:26:57.450854 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:26:57.450865 3798194 disk_worker.go:723] disk space is not compliant W1121 11:26:57.450868 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:26:57.459232 3798194 disk_worker.go:509] system disk:vda1 I1121 11:26:57.459272 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:26:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:26:59.450103 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:26:59.450187 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:26:59.450214 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:26:59.470684 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:27:06.412874 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:27:06.412898 3798194 memory.go:184] no items to output this cycle I1121 11:27:06.413045 3798194 cpu.go:275] no items to output this cycle E1121 11:27:16.413008 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:27:16.413031 3798194 memory.go:184] no items to output this cycle I1121 11:27:16.413235 3798194 cpu.go:275] no items to output this cycle I1121 11:27:17.450700 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:27:17.450709 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:27:26.411516 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:27:26.411547 3798194 memory.go:191] Add success. I1121 11:27:26.411874 3798194 cpu.go:282] Add success. I1121 11:27:26.441527 3798194 net.go:642] Add success. I1121 11:27:26.445531 3798194 net.go:778] primary dev: ETH0 I1121 11:27:26.445548 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:27:26.445577 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:27:29.450029 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:27:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:27:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:27:36.412534 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:27:36.412556 3798194 memory.go:184] no items to output this cycle I1121 11:27:36.412696 3798194 cpu.go:275] no items to output this cycle E1121 11:27:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:27:46.413253 3798194 memory.go:184] no items to output this cycle I1121 11:27:46.413398 3798194 cpu.go:275] no items to output this cycle E1121 11:27:56.412845 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:27:56.412879 3798194 memory.go:191] Add success. W1121 11:27:56.412912 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:27:56.412924 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:27:56.412927 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:27:56.417353 3798194 cpu.go:282] Add success. I1121 11:27:56.441862 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:27:56.442458 3798194 net.go:642] Add success. I1121 11:27:56.444169 3798194 net.go:778] primary dev: ETH0 I1121 11:27:56.444184 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:27:56.444196 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:27:56.726149 3798194 disk_info.go:125] begin check local disk info of client I1121 11:27:56.730441 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:27:56.730450 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003be1c0 0xc0003be200] I1121 11:27:57.450734 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:27:57.450947 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:27:57.450958 3798194 disk_worker.go:723] disk space is not compliant W1121 11:27:57.450962 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:27:57.459921 3798194 disk_worker.go:509] system disk:vda1 I1121 11:27:57.459965 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:27:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:27:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:27:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:27:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:27:59.471650 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:28:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:28:06.413259 3798194 memory.go:184] no items to output this cycle I1121 11:28:06.413403 3798194 cpu.go:275] no items to output this cycle E1121 11:28:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:28:16.411981 3798194 memory.go:184] no items to output this cycle I1121 11:28:16.412135 3798194 cpu.go:275] no items to output this cycle E1121 11:28:26.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:28:26.412273 3798194 memory.go:191] Add success. I1121 11:28:26.416657 3798194 cpu.go:282] Add success. I1121 11:28:26.441266 3798194 net.go:642] Add success. I1121 11:28:26.442744 3798194 net.go:778] primary dev: ETH0 I1121 11:28:26.442760 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:28:26.442775 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:28:29.450032 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:28:29.450144 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:28:29.450171 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:28:36.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:28:36.411967 3798194 memory.go:184] no items to output this cycle I1121 11:28:36.412105 3798194 cpu.go:275] no items to output this cycle E1121 11:28:46.413025 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:28:46.413053 3798194 memory.go:184] no items to output this cycle I1121 11:28:46.413227 3798194 cpu.go:275] no items to output this cycle E1121 11:28:56.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:28:56.412272 3798194 memory.go:191] Add success. W1121 11:28:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:28:56.412314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:28:56.412317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:28:56.416811 3798194 cpu.go:282] Add success. I1121 11:28:56.441553 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:28:56.442594 3798194 net.go:642] Add success. I1121 11:28:56.444054 3798194 net.go:778] primary dev: ETH0 I1121 11:28:56.444070 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:28:56.444083 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:28:56.730554 3798194 disk_info.go:125] begin check local disk info of client I1121 11:28:56.734520 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:28:56.734528 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047e080 0xc00047e0c0] I1121 11:28:57.450680 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:28:57.450863 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:28:57.450919 3798194 disk_worker.go:723] disk space is not compliant W1121 11:28:57.450922 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:28:57.459792 3798194 disk_worker.go:509] system disk:vda1 I1121 11:28:57.459844 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:28:58.446371 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:28:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:28:59.450454 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:28:59.450480 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:28:59.471010 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:29:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:29:06.411985 3798194 memory.go:184] no items to output this cycle I1121 11:29:06.412143 3798194 cpu.go:275] no items to output this cycle E1121 11:29:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:29:16.411974 3798194 memory.go:184] no items to output this cycle I1121 11:29:16.412133 3798194 cpu.go:275] no items to output this cycle E1121 11:29:26.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:29:26.411945 3798194 memory.go:191] Add success. I1121 11:29:26.412311 3798194 cpu.go:282] Add success. I1121 11:29:26.442039 3798194 net.go:642] Add success. I1121 11:29:26.449861 3798194 net.go:778] primary dev: ETH0 I1121 11:29:26.449876 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:29:26.449891 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:29:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:29:29.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:29:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:29:36.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:29:36.413252 3798194 memory.go:184] no items to output this cycle I1121 11:29:36.413399 3798194 cpu.go:275] no items to output this cycle E1121 11:29:46.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:29:46.413254 3798194 memory.go:184] no items to output this cycle I1121 11:29:46.413401 3798194 cpu.go:275] no items to output this cycle E1121 11:29:56.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:29:56.412887 3798194 memory.go:191] Add success. W1121 11:29:56.412916 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:29:56.412928 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:29:56.412931 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:29:56.417356 3798194 cpu.go:282] Add success. I1121 11:29:56.441862 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:29:56.442385 3798194 net.go:642] Add success. I1121 11:29:56.443957 3798194 net.go:778] primary dev: ETH0 I1121 11:29:56.443971 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:29:56.443983 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:29:56.734624 3798194 disk_info.go:125] begin check local disk info of client I1121 11:29:56.738579 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:29:56.738587 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ccf40 0xc0003ccf80] I1121 11:29:57.450566 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:29:57.450769 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:29:57.450780 3798194 disk_worker.go:723] disk space is not compliant W1121 11:29:57.450783 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:29:57.459097 3798194 disk_worker.go:509] system disk:vda1 I1121 11:29:57.459153 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:29:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:29:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:29:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:29:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:29:59.471647 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:30:06.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:30:06.411973 3798194 memory.go:184] no items to output this cycle I1121 11:30:06.412336 3798194 cpu.go:275] no items to output this cycle E1121 11:30:16.412516 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:30:16.412539 3798194 memory.go:184] no items to output this cycle I1121 11:30:16.412680 3798194 cpu.go:275] no items to output this cycle I1121 11:30:17.451590 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:30:17.451598 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:30:26.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:30:26.412888 3798194 memory.go:191] Add success. I1121 11:30:26.417246 3798194 cpu.go:282] Add success. I1121 11:30:26.441764 3798194 net.go:642] Add success. I1121 11:30:26.443303 3798194 net.go:778] primary dev: ETH0 I1121 11:30:26.443318 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:30:26.443331 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:30:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:30:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:30:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:30:36.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:30:36.412248 3798194 memory.go:184] no items to output this cycle I1121 11:30:36.412391 3798194 cpu.go:275] no items to output this cycle E1121 11:30:46.413451 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:30:46.413472 3798194 memory.go:184] no items to output this cycle I1121 11:30:46.413621 3798194 cpu.go:275] no items to output this cycle E1121 11:30:56.411909 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:30:56.411942 3798194 memory.go:191] Add success. W1121 11:30:56.411974 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:30:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:30:56.411990 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:30:56.412341 3798194 cpu.go:282] Add success. I1121 11:30:56.444632 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:30:56.445134 3798194 net.go:642] Add success. I1121 11:30:56.448882 3798194 net.go:778] primary dev: ETH0 I1121 11:30:56.448896 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:30:56.448910 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:30:56.738691 3798194 disk_info.go:125] begin check local disk info of client I1121 11:30:56.742625 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:30:56.742634 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00035b640 0xc00035b680] E1121 11:30:57.450572 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 11:30:57.450585 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:30:57.450798 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:30:57.450808 3798194 disk_worker.go:723] disk space is not compliant W1121 11:30:57.450811 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:30:57.459288 3798194 disk_worker.go:509] system disk:vda1 I1121 11:30:57.459333 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 11:30:58.448055 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 11:30:58.448070 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 11:30:59.448607 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 11:30:59.449682 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:30:59.449732 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:30:59.449753 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:30:59.471183 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:31:06.412782 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:31:06.412806 3798194 memory.go:184] no items to output this cycle I1121 11:31:06.412953 3798194 cpu.go:275] no items to output this cycle E1121 11:31:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:31:16.411974 3798194 memory.go:184] no items to output this cycle I1121 11:31:16.412128 3798194 cpu.go:275] no items to output this cycle E1121 11:31:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:31:26.411961 3798194 memory.go:191] Add success. I1121 11:31:26.412309 3798194 cpu.go:282] Add success. I1121 11:31:26.442037 3798194 net.go:642] Add success. I1121 11:31:26.445897 3798194 net.go:778] primary dev: ETH0 I1121 11:31:26.445912 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:31:26.445925 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:31:29.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:31:29.450149 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:31:29.450174 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:31:36.413329 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:31:36.413365 3798194 memory.go:184] no items to output this cycle I1121 11:31:36.413650 3798194 cpu.go:275] no items to output this cycle E1121 11:31:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:31:46.413260 3798194 memory.go:184] no items to output this cycle I1121 11:31:46.413408 3798194 cpu.go:275] no items to output this cycle E1121 11:31:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:31:56.411951 3798194 memory.go:191] Add success. W1121 11:31:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:31:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:31:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:31:56.412370 3798194 cpu.go:282] Add success. I1121 11:31:56.441838 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:31:56.442356 3798194 net.go:642] Add success. I1121 11:31:56.446606 3798194 net.go:778] primary dev: ETH0 I1121 11:31:56.446621 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:31:56.446633 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:31:56.742737 3798194 disk_info.go:125] begin check local disk info of client I1121 11:31:56.747061 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:31:56.747070 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037d1c0 0xc00037d200] I1121 11:31:57.450794 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:31:57.450960 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:31:57.451015 3798194 disk_worker.go:723] disk space is not compliant W1121 11:31:57.451018 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:31:57.460285 3798194 disk_worker.go:509] system disk:vda1 I1121 11:31:57.460328 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:31:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:31:59.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:31:59.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:31:59.450501 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:31:59.471012 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:32:06.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:32:06.412876 3798194 memory.go:184] no items to output this cycle I1121 11:32:06.413022 3798194 cpu.go:275] no items to output this cycle E1121 11:32:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:32:16.411987 3798194 memory.go:184] no items to output this cycle I1121 11:32:16.412148 3798194 cpu.go:275] no items to output this cycle E1121 11:32:26.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:32:26.411945 3798194 memory.go:191] Add success. I1121 11:32:26.412298 3798194 cpu.go:282] Add success. I1121 11:32:26.442017 3798194 net.go:642] Add success. I1121 11:32:26.445981 3798194 net.go:778] primary dev: ETH0 I1121 11:32:26.445998 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:32:26.446013 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:32:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:32:29.450462 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:32:29.450489 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:32:36.412261 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:32:36.412285 3798194 memory.go:184] no items to output this cycle I1121 11:32:36.412431 3798194 cpu.go:275] no items to output this cycle E1121 11:32:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:32:46.411978 3798194 memory.go:184] no items to output this cycle I1121 11:32:46.412139 3798194 cpu.go:275] no items to output this cycle E1121 11:32:56.413006 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:32:56.413040 3798194 memory.go:191] Add success. W1121 11:32:56.413068 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:32:56.413081 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:32:56.413084 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:32:56.417626 3798194 cpu.go:282] Add success. I1121 11:32:56.441124 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:32:56.441608 3798194 net.go:642] Add success. I1121 11:32:56.443229 3798194 net.go:778] primary dev: ETH0 I1121 11:32:56.443246 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:32:56.443261 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:32:56.747162 3798194 disk_info.go:125] begin check local disk info of client I1121 11:32:56.751096 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:32:56.751105 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046ae80 0xc00046aec0] I1121 11:32:57.451009 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:32:57.451209 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:32:57.451266 3798194 disk_worker.go:723] disk space is not compliant W1121 11:32:57.451270 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:32:57.459727 3798194 disk_worker.go:509] system disk:vda1 I1121 11:32:57.459769 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:32:58.446298 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:32:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:32:59.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:32:59.450503 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:32:59.471011 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:33:06.413198 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:33:06.413220 3798194 memory.go:184] no items to output this cycle I1121 11:33:06.413379 3798194 cpu.go:275] no items to output this cycle E1121 11:33:16.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:33:16.412252 3798194 memory.go:184] no items to output this cycle I1121 11:33:16.412406 3798194 cpu.go:275] no items to output this cycle I1121 11:33:17.452095 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:33:17.452106 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:33:26.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:33:26.412898 3798194 memory.go:191] Add success. I1121 11:33:26.417457 3798194 cpu.go:282] Add success. I1121 11:33:26.442218 3798194 net.go:642] Add success. I1121 11:33:26.443719 3798194 net.go:778] primary dev: ETH0 I1121 11:33:26.443734 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:33:26.443746 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:33:29.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:33:29.450465 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:33:29.450490 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:33:36.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:33:36.411973 3798194 memory.go:184] no items to output this cycle I1121 11:33:36.412126 3798194 cpu.go:275] no items to output this cycle E1121 11:33:46.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:33:46.413255 3798194 memory.go:184] no items to output this cycle I1121 11:33:46.413402 3798194 cpu.go:275] no items to output this cycle E1121 11:33:56.413362 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:33:56.413397 3798194 memory.go:191] Add success. W1121 11:33:56.413424 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:33:56.413435 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:33:56.413439 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:33:56.417889 3798194 cpu.go:282] Add success. I1121 11:33:56.441248 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:33:56.441749 3798194 net.go:642] Add success. I1121 11:33:56.443504 3798194 net.go:778] primary dev: ETH0 I1121 11:33:56.443527 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:33:56.443542 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:33:56.752216 3798194 disk_info.go:125] begin check local disk info of client I1121 11:33:56.756348 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:33:56.756357 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f4680 0xc0003f46c0] I1121 11:33:57.445887 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:33:57.446093 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:33:57.446169 3798194 disk_worker.go:723] disk space is not compliant W1121 11:33:57.446172 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:33:57.448266 3798194 disk_worker.go:509] system disk:vda1 I1121 11:33:57.448306 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:33:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:33:59.450007 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:33:59.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:33:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:33:59.471643 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:34:06.412255 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:34:06.412283 3798194 memory.go:184] no items to output this cycle I1121 11:34:06.412440 3798194 cpu.go:275] no items to output this cycle E1121 11:34:16.413503 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:34:16.413541 3798194 memory.go:184] no items to output this cycle I1121 11:34:16.413830 3798194 cpu.go:275] no items to output this cycle E1121 11:34:26.411247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:34:26.411282 3798194 memory.go:191] Add success. I1121 11:34:26.415550 3798194 cpu.go:282] Add success. I1121 11:34:26.442063 3798194 net.go:642] Add success. I1121 11:34:26.443611 3798194 net.go:778] primary dev: ETH0 I1121 11:34:26.443626 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:34:26.443640 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:34:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:34:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:34:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:34:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:34:36.413254 3798194 memory.go:184] no items to output this cycle I1121 11:34:36.413395 3798194 cpu.go:275] no items to output this cycle E1121 11:34:46.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:34:46.412259 3798194 memory.go:184] no items to output this cycle I1121 11:34:46.412403 3798194 cpu.go:275] no items to output this cycle E1121 11:34:56.413107 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:34:56.413154 3798194 memory.go:191] Add success. W1121 11:34:56.413188 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:34:56.413201 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:34:56.413205 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:34:56.417680 3798194 cpu.go:282] Add success. I1121 11:34:56.440955 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:34:56.441535 3798194 net.go:642] Add success. I1121 11:34:56.443124 3798194 net.go:778] primary dev: ETH0 I1121 11:34:56.443141 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:34:56.443153 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:34:56.756465 3798194 disk_info.go:125] begin check local disk info of client I1121 11:34:56.760492 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:34:56.760501 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030c700 0xc00030c740] I1121 11:34:57.451024 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:34:57.451256 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:34:57.451267 3798194 disk_worker.go:723] disk space is not compliant W1121 11:34:57.451270 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:34:57.460068 3798194 disk_worker.go:509] system disk:vda1 I1121 11:34:57.460124 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:34:58.445696 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:34:59.449674 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:34:59.449752 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:34:59.449780 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:34:59.471323 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:35:06.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:35:06.412260 3798194 memory.go:184] no items to output this cycle I1121 11:35:06.412395 3798194 cpu.go:275] no items to output this cycle E1121 11:35:16.412775 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:35:16.412798 3798194 memory.go:184] no items to output this cycle I1121 11:35:16.412945 3798194 cpu.go:275] no items to output this cycle E1121 11:35:26.412868 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:35:26.412906 3798194 memory.go:191] Add success. I1121 11:35:26.417328 3798194 cpu.go:282] Add success. I1121 11:35:26.441746 3798194 net.go:642] Add success. I1121 11:35:26.443197 3798194 net.go:778] primary dev: ETH0 I1121 11:35:26.443211 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:35:26.443225 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:35:29.449978 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:35:29.450055 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:35:29.450081 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:35:36.412923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:35:36.412945 3798194 memory.go:184] no items to output this cycle I1121 11:35:36.413097 3798194 cpu.go:275] no items to output this cycle E1121 11:35:46.412430 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:35:46.412459 3798194 memory.go:184] no items to output this cycle I1121 11:35:46.412614 3798194 cpu.go:275] no items to output this cycle E1121 11:35:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:35:56.411948 3798194 memory.go:191] Add success. W1121 11:35:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:35:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:35:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:35:56.412350 3798194 cpu.go:282] Add success. I1121 11:35:56.441580 3798194 event_worker.go:152] Polling the log file for events... I1121 11:35:56.444878 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:35:56.445391 3798194 net.go:642] Add success. I1121 11:35:56.449439 3798194 net.go:778] primary dev: ETH0 I1121 11:35:56.449454 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:35:56.449467 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:35:56.449485 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795230356, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 11:35:56.457338 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 I1121 11:35:56.760606 3798194 disk_info.go:125] begin check local disk info of client I1121 11:35:56.764681 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:35:56.764690 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036eb40 0xc00036eb80] E1121 11:35:57.445584 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 11:35:57.445602 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:35:57.445799 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:35:57.445874 3798194 disk_worker.go:723] disk space is not compliant W1121 11:35:57.445878 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:35:57.450818 3798194 disk_worker.go:509] system disk:vda1 I1121 11:35:57.450870 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 11:35:58.448103 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 11:35:58.448133 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 11:35:59.448628 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 11:35:59.449657 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:35:59.449714 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:35:59.449738 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:35:59.471194 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:36:06.412830 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:36:06.412854 3798194 memory.go:184] no items to output this cycle I1121 11:36:06.413000 3798194 cpu.go:275] no items to output this cycle E1121 11:36:16.412333 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:36:16.412353 3798194 memory.go:184] no items to output this cycle I1121 11:36:16.412506 3798194 cpu.go:275] no items to output this cycle I1121 11:36:17.452478 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:36:17.452487 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:36:26.411293 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:36:26.411327 3798194 memory.go:191] Add success. I1121 11:36:26.411663 3798194 cpu.go:282] Add success. I1121 11:36:26.441250 3798194 net.go:642] Add success. I1121 11:36:26.445492 3798194 net.go:778] primary dev: ETH0 I1121 11:36:26.445511 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:36:26.445525 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:36:29.450316 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:36:29.450404 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:36:29.450429 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:36:36.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:36:36.413259 3798194 memory.go:184] no items to output this cycle I1121 11:36:36.413407 3798194 cpu.go:275] no items to output this cycle E1121 11:36:46.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:36:46.411993 3798194 memory.go:184] no items to output this cycle I1121 11:36:46.412157 3798194 cpu.go:275] no items to output this cycle E1121 11:36:56.411896 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:36:56.411928 3798194 memory.go:191] Add success. W1121 11:36:56.411956 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:36:56.411968 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:36:56.411972 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:36:56.412340 3798194 cpu.go:282] Add success. I1121 11:36:56.441813 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:36:56.442335 3798194 net.go:642] Add success. I1121 11:36:56.446838 3798194 net.go:778] primary dev: ETH0 I1121 11:36:56.446852 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:36:56.446865 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:36:56.764796 3798194 disk_info.go:125] begin check local disk info of client I1121 11:36:56.768783 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:36:56.768792 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047e0c0 0xc00047e100] I1121 11:36:57.445584 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:36:57.445856 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:36:57.445867 3798194 disk_worker.go:723] disk space is not compliant W1121 11:36:57.445870 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:36:57.450827 3798194 disk_worker.go:509] system disk:vda1 I1121 11:36:57.450867 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:36:58.446370 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:36:59.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:36:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:36:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:36:59.470689 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:37:06.412526 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:37:06.412555 3798194 memory.go:184] no items to output this cycle I1121 11:37:06.412707 3798194 cpu.go:275] no items to output this cycle E1121 11:37:16.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:37:16.412270 3798194 memory.go:184] no items to output this cycle I1121 11:37:16.412419 3798194 cpu.go:275] no items to output this cycle E1121 11:37:26.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:37:26.412893 3798194 memory.go:191] Add success. I1121 11:37:26.417060 3798194 cpu.go:282] Add success. I1121 11:37:26.441620 3798194 net.go:642] Add success. I1121 11:37:26.443283 3798194 net.go:778] primary dev: ETH0 I1121 11:37:26.443298 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:37:26.443311 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:37:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:37:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:37:29.450149 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:37:36.411787 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:37:36.411811 3798194 memory.go:184] no items to output this cycle I1121 11:37:36.411964 3798194 cpu.go:275] no items to output this cycle E1121 11:37:46.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:37:46.412875 3798194 memory.go:184] no items to output this cycle I1121 11:37:46.413021 3798194 cpu.go:275] no items to output this cycle E1121 11:37:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:37:56.411948 3798194 memory.go:191] Add success. W1121 11:37:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:37:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:37:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:37:56.412320 3798194 cpu.go:282] Add success. I1121 11:37:56.441767 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:37:56.442270 3798194 net.go:642] Add success. I1121 11:37:56.446687 3798194 net.go:778] primary dev: ETH0 I1121 11:37:56.446703 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:37:56.446715 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:37:56.769138 3798194 disk_info.go:125] begin check local disk info of client I1121 11:37:56.773150 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:37:56.773159 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003cb740 0xc0003cb780] I1121 11:37:57.451029 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:37:57.451221 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:37:57.451279 3798194 disk_worker.go:723] disk space is not compliant W1121 11:37:57.451283 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:37:57.459816 3798194 disk_worker.go:509] system disk:vda1 I1121 11:37:57.459858 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:37:58.446460 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:37:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:37:59.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:37:59.450128 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:37:59.471644 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:38:06.412091 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:38:06.412129 3798194 memory.go:184] no items to output this cycle I1121 11:38:06.412286 3798194 cpu.go:275] no items to output this cycle E1121 11:38:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:38:16.413254 3798194 memory.go:184] no items to output this cycle I1121 11:38:16.413602 3798194 cpu.go:275] no items to output this cycle E1121 11:38:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:38:26.411953 3798194 memory.go:191] Add success. I1121 11:38:26.412337 3798194 cpu.go:282] Add success. I1121 11:38:26.441939 3798194 net.go:642] Add success. I1121 11:38:26.446058 3798194 net.go:778] primary dev: ETH0 I1121 11:38:26.446073 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:38:26.446087 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:38:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:38:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:38:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:38:36.412819 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:38:36.412848 3798194 memory.go:184] no items to output this cycle I1121 11:38:36.412990 3798194 cpu.go:275] no items to output this cycle E1121 11:38:46.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:38:46.412260 3798194 memory.go:184] no items to output this cycle I1121 11:38:46.412405 3798194 cpu.go:275] no items to output this cycle E1121 11:38:56.411459 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:38:56.411490 3798194 memory.go:191] Add success. W1121 11:38:56.411524 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:38:56.411536 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:38:56.411541 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:38:56.416163 3798194 cpu.go:282] Add success. I1121 11:38:56.441434 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:38:56.441910 3798194 net.go:642] Add success. I1121 11:38:56.443577 3798194 net.go:778] primary dev: ETH0 I1121 11:38:56.443593 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:38:56.443606 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:38:56.773260 3798194 disk_info.go:125] begin check local disk info of client I1121 11:38:56.778818 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:38:56.778827 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002f4340 0xc0002f4380] I1121 11:38:57.450743 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:38:57.450945 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:38:57.450956 3798194 disk_worker.go:723] disk space is not compliant W1121 11:38:57.450960 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:38:57.462492 3798194 disk_worker.go:509] system disk:vda1 I1121 11:38:57.462536 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:38:58.446377 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:38:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:38:59.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:38:59.450146 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:38:59.471700 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:39:06.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:39:06.411973 3798194 memory.go:184] no items to output this cycle I1121 11:39:06.412124 3798194 cpu.go:275] no items to output this cycle E1121 11:39:16.412813 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:39:16.412841 3798194 memory.go:184] no items to output this cycle I1121 11:39:16.412988 3798194 cpu.go:275] no items to output this cycle I1121 11:39:17.453119 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:39:17.453129 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:39:26.411293 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:39:26.411327 3798194 memory.go:191] Add success. I1121 11:39:26.411669 3798194 cpu.go:282] Add success. I1121 11:39:26.441272 3798194 net.go:642] Add success. I1121 11:39:26.445783 3798194 net.go:778] primary dev: ETH0 I1121 11:39:26.445798 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:39:26.445812 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:39:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:39:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:39:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:39:36.413164 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:39:36.413185 3798194 memory.go:184] no items to output this cycle I1121 11:39:36.413327 3798194 cpu.go:275] no items to output this cycle E1121 11:39:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:39:46.411976 3798194 memory.go:184] no items to output this cycle I1121 11:39:46.412362 3798194 cpu.go:275] no items to output this cycle E1121 11:39:56.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:39:56.412891 3798194 memory.go:191] Add success. W1121 11:39:56.412920 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:39:56.412932 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:39:56.412935 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:39:56.417127 3798194 cpu.go:282] Add success. I1121 11:39:56.441695 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:39:56.442293 3798194 net.go:642] Add success. I1121 11:39:56.443819 3798194 net.go:778] primary dev: ETH0 I1121 11:39:56.443833 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:39:56.443846 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:39:56.780142 3798194 disk_info.go:125] begin check local disk info of client I1121 11:39:56.784006 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:39:56.784016 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ea480 0xc0003ea4c0] I1121 11:39:57.450941 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:39:57.451098 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:39:57.451174 3798194 disk_worker.go:723] disk space is not compliant W1121 11:39:57.451177 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:39:57.460005 3798194 disk_worker.go:509] system disk:vda1 I1121 11:39:57.460045 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:39:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:39:59.450108 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:39:59.450201 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:39:59.450228 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:39:59.470702 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:40:06.412221 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:40:06.412243 3798194 memory.go:184] no items to output this cycle I1121 11:40:06.412389 3798194 cpu.go:275] no items to output this cycle E1121 11:40:16.413102 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:40:16.413139 3798194 memory.go:184] no items to output this cycle I1121 11:40:16.413287 3798194 cpu.go:275] no items to output this cycle E1121 11:40:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:40:26.411957 3798194 memory.go:191] Add success. I1121 11:40:26.412310 3798194 cpu.go:282] Add success. I1121 11:40:26.441898 3798194 net.go:642] Add success. I1121 11:40:26.445764 3798194 net.go:778] primary dev: ETH0 I1121 11:40:26.445779 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:40:26.445792 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:40:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:40:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:40:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:40:36.413158 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:40:36.413181 3798194 memory.go:184] no items to output this cycle I1121 11:40:36.413332 3798194 cpu.go:275] no items to output this cycle E1121 11:40:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:40:46.411973 3798194 memory.go:184] no items to output this cycle I1121 11:40:46.412124 3798194 cpu.go:275] no items to output this cycle E1121 11:40:56.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:40:56.413265 3798194 memory.go:191] Add success. W1121 11:40:56.413291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:40:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:40:56.413306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:40:56.417907 3798194 cpu.go:282] Add success. I1121 11:40:56.444212 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:40:56.444696 3798194 net.go:642] Add success. I1121 11:40:56.446307 3798194 net.go:778] primary dev: ETH0 I1121 11:40:56.446323 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:40:56.446337 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:40:56.784117 3798194 disk_info.go:125] begin check local disk info of client I1121 11:40:56.787909 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:40:56.787918 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004497c0 0xc000449800] E1121 11:40:57.450862 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 11:40:57.450876 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:40:57.451034 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:40:57.451083 3798194 disk_worker.go:723] disk space is not compliant W1121 11:40:57.451086 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:40:57.460541 3798194 disk_worker.go:509] system disk:vda1 I1121 11:40:57.460578 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 11:40:58.448405 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 11:40:58.448419 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 11:40:59.449105 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 11:40:59.450198 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:40:59.450252 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:40:59.450270 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:40:59.470728 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:41:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:41:06.413257 3798194 memory.go:184] no items to output this cycle I1121 11:41:06.413402 3798194 cpu.go:275] no items to output this cycle E1121 11:41:16.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:41:16.412889 3798194 memory.go:184] no items to output this cycle I1121 11:41:16.413034 3798194 cpu.go:275] no items to output this cycle E1121 11:41:26.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:41:26.413282 3798194 memory.go:191] Add success. I1121 11:41:26.417719 3798194 cpu.go:282] Add success. I1121 11:41:26.441388 3798194 net.go:642] Add success. I1121 11:41:26.442823 3798194 net.go:778] primary dev: ETH0 I1121 11:41:26.442837 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:41:26.442851 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:41:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:41:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:41:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:41:36.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:41:36.413265 3798194 memory.go:184] no items to output this cycle I1121 11:41:36.413411 3798194 cpu.go:275] no items to output this cycle E1121 11:41:46.411945 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:41:46.411966 3798194 memory.go:184] no items to output this cycle I1121 11:41:46.412129 3798194 cpu.go:275] no items to output this cycle E1121 11:41:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:41:56.413265 3798194 memory.go:191] Add success. W1121 11:41:56.413291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:41:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:41:56.413308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:41:56.417548 3798194 cpu.go:282] Add success. I1121 11:41:56.440954 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:41:56.441547 3798194 net.go:642] Add success. I1121 11:41:56.443219 3798194 net.go:778] primary dev: ETH0 I1121 11:41:56.443234 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:41:56.443246 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:41:56.788022 3798194 disk_info.go:125] begin check local disk info of client I1121 11:41:56.791877 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:41:56.791887 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a2980 0xc0003a29c0] I1121 11:41:57.450513 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:41:57.450709 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:41:57.450719 3798194 disk_worker.go:723] disk space is not compliant W1121 11:41:57.450722 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:41:57.459046 3798194 disk_worker.go:509] system disk:vda1 I1121 11:41:57.459088 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:41:58.445993 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:41:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:41:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:41:59.450133 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:41:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:42:06.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:42:06.412248 3798194 memory.go:184] no items to output this cycle I1121 11:42:06.412394 3798194 cpu.go:275] no items to output this cycle E1121 11:42:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:42:16.411980 3798194 memory.go:184] no items to output this cycle I1121 11:42:16.412140 3798194 cpu.go:275] no items to output this cycle I1121 11:42:17.453933 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:42:17.453941 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:42:26.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:42:26.412886 3798194 memory.go:191] Add success. I1121 11:42:26.417164 3798194 cpu.go:282] Add success. I1121 11:42:26.441536 3798194 net.go:642] Add success. I1121 11:42:26.442993 3798194 net.go:778] primary dev: ETH0 I1121 11:42:26.443007 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:42:26.443020 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:42:29.450008 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:42:29.450083 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:42:29.450122 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:42:36.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:42:36.411970 3798194 memory.go:184] no items to output this cycle I1121 11:42:36.412135 3798194 cpu.go:275] no items to output this cycle E1121 11:42:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:42:46.413254 3798194 memory.go:184] no items to output this cycle I1121 11:42:46.413395 3798194 cpu.go:275] no items to output this cycle E1121 11:42:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:42:56.411956 3798194 memory.go:191] Add success. W1121 11:42:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:42:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:42:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:42:56.412350 3798194 cpu.go:282] Add success. I1121 11:42:56.441237 3798194 net.go:642] Add success. I1121 11:42:56.445734 3798194 net.go:778] primary dev: ETH0 I1121 11:42:56.445751 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:42:56.445765 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:42:56.441571 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:42:56.791994 3798194 disk_info.go:125] begin check local disk info of client I1121 11:42:56.795942 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:42:56.795952 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00025e4c0 0xc00025e500] I1121 11:42:57.445890 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:42:57.446091 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:42:57.446165 3798194 disk_worker.go:723] disk space is not compliant W1121 11:42:57.446168 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:42:57.448269 3798194 disk_worker.go:509] system disk:vda1 I1121 11:42:57.448310 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:42:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:42:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:42:59.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:42:59.450502 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:42:59.471015 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:43:06.413647 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:43:06.413673 3798194 memory.go:184] no items to output this cycle I1121 11:43:06.413817 3798194 cpu.go:275] no items to output this cycle E1121 11:43:16.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:43:16.411986 3798194 memory.go:184] no items to output this cycle I1121 11:43:16.412141 3798194 cpu.go:275] no items to output this cycle E1121 11:43:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:43:26.413269 3798194 memory.go:191] Add success. I1121 11:43:26.417645 3798194 cpu.go:282] Add success. I1121 11:43:26.441291 3798194 net.go:642] Add success. I1121 11:43:26.442937 3798194 net.go:778] primary dev: ETH0 I1121 11:43:26.442951 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:43:26.442964 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:43:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:43:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:43:29.450124 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:43:36.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:43:36.413249 3798194 memory.go:184] no items to output this cycle I1121 11:43:36.413395 3798194 cpu.go:275] no items to output this cycle E1121 11:43:46.413565 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:43:46.413589 3798194 memory.go:184] no items to output this cycle I1121 11:43:46.413730 3798194 cpu.go:275] no items to output this cycle E1121 11:43:56.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:43:56.413267 3798194 memory.go:191] Add success. W1121 11:43:56.413292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:43:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:43:56.413308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:43:56.417872 3798194 cpu.go:282] Add success. I1121 11:43:56.440944 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:43:56.441499 3798194 net.go:642] Add success. I1121 11:43:56.442943 3798194 net.go:778] primary dev: ETH0 I1121 11:43:56.442957 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:43:56.442970 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:43:56.797142 3798194 disk_info.go:125] begin check local disk info of client I1121 11:43:56.801038 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:43:56.801048 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071bc0 0xc000071c00] I1121 11:43:57.450962 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:43:57.451148 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:43:57.451203 3798194 disk_worker.go:723] disk space is not compliant W1121 11:43:57.451206 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:43:57.459955 3798194 disk_worker.go:509] system disk:vda1 I1121 11:43:57.459995 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:43:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:43:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:43:59.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:43:59.450496 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:43:59.471009 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:44:06.413249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:44:06.413270 3798194 memory.go:184] no items to output this cycle I1121 11:44:06.413413 3798194 cpu.go:275] no items to output this cycle E1121 11:44:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:44:16.412258 3798194 memory.go:184] no items to output this cycle I1121 11:44:16.412420 3798194 cpu.go:275] no items to output this cycle E1121 11:44:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:44:26.411950 3798194 memory.go:191] Add success. I1121 11:44:26.412293 3798194 cpu.go:282] Add success. I1121 11:44:26.441897 3798194 net.go:642] Add success. I1121 11:44:26.445638 3798194 net.go:778] primary dev: ETH0 I1121 11:44:26.445654 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:44:26.445668 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:44:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:44:29.450133 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:44:29.450163 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:44:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:44:36.412254 3798194 memory.go:184] no items to output this cycle I1121 11:44:36.412393 3798194 cpu.go:275] no items to output this cycle E1121 11:44:46.413205 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:44:46.413232 3798194 memory.go:184] no items to output this cycle I1121 11:44:46.413378 3798194 cpu.go:275] no items to output this cycle E1121 11:44:56.413256 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:44:56.413290 3798194 memory.go:191] Add success. W1121 11:44:56.413317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:44:56.413328 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:44:56.413332 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:44:56.417495 3798194 cpu.go:282] Add success. I1121 11:44:56.440952 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:44:56.441526 3798194 net.go:642] Add success. I1121 11:44:56.443139 3798194 net.go:778] primary dev: ETH0 I1121 11:44:56.443154 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:44:56.443172 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:44:56.802134 3798194 disk_info.go:125] begin check local disk info of client I1121 11:44:56.806099 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:44:56.806108 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fa300 0xc0003fa340] I1121 11:44:57.450943 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:44:57.451103 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:44:57.451180 3798194 disk_worker.go:723] disk space is not compliant W1121 11:44:57.451183 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:44:57.459923 3798194 disk_worker.go:509] system disk:vda1 I1121 11:44:57.459966 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:44:58.446012 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:44:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:44:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:44:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:44:59.471688 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:45:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:45:06.411983 3798194 memory.go:184] no items to output this cycle I1121 11:45:06.412137 3798194 cpu.go:275] no items to output this cycle E1121 11:45:16.412843 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:45:16.412871 3798194 memory.go:184] no items to output this cycle I1121 11:45:16.413016 3798194 cpu.go:275] no items to output this cycle I1121 11:45:17.454493 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:45:17.454502 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:45:26.412835 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:45:26.412866 3798194 memory.go:191] Add success. I1121 11:45:26.417233 3798194 cpu.go:282] Add success. I1121 11:45:26.441835 3798194 net.go:642] Add success. I1121 11:45:26.443322 3798194 net.go:778] primary dev: ETH0 I1121 11:45:26.443336 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:45:26.443348 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:45:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:45:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:45:29.450512 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:45:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:45:36.412256 3798194 memory.go:184] no items to output this cycle I1121 11:45:36.412398 3798194 cpu.go:275] no items to output this cycle E1121 11:45:46.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:45:46.411968 3798194 memory.go:184] no items to output this cycle I1121 11:45:46.412126 3798194 cpu.go:275] no items to output this cycle E1121 11:45:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:45:56.411951 3798194 memory.go:191] Add success. W1121 11:45:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:45:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:45:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:45:56.412364 3798194 cpu.go:282] Add success. I1121 11:45:56.441569 3798194 event_worker.go:152] Polling the log file for events... I1121 11:45:56.444715 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:45:56.445229 3798194 net.go:642] Add success. I1121 11:45:56.449322 3798194 net.go:778] primary dev: ETH0 I1121 11:45:56.449337 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:45:56.449352 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:45:56.807146 3798194 disk_info.go:125] begin check local disk info of client I1121 11:45:56.811087 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:45:56.811096 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000334a80 0xc000334ac0] E1121 11:45:57.450516 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 11:45:57.450530 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:45:57.450719 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:45:57.450730 3798194 disk_worker.go:723] disk space is not compliant W1121 11:45:57.450733 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:45:57.459121 3798194 disk_worker.go:509] system disk:vda1 I1121 11:45:57.459164 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 11:45:58.464637 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 11:45:58.464653 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 11:45:59.448982 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 11:45:59.450055 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:45:59.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:45:59.450140 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:45:59.471647 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:46:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:46:06.412269 3798194 memory.go:184] no items to output this cycle I1121 11:46:06.412410 3798194 cpu.go:275] no items to output this cycle E1121 11:46:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:46:16.411974 3798194 memory.go:184] no items to output this cycle I1121 11:46:16.412137 3798194 cpu.go:275] no items to output this cycle E1121 11:46:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:46:26.411947 3798194 memory.go:191] Add success. I1121 11:46:26.412305 3798194 cpu.go:282] Add success. I1121 11:46:26.441877 3798194 net.go:642] Add success. I1121 11:46:26.446070 3798194 net.go:778] primary dev: ETH0 I1121 11:46:26.446086 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:46:26.446099 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:46:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:46:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:46:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:46:36.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:46:36.413251 3798194 memory.go:184] no items to output this cycle I1121 11:46:36.413401 3798194 cpu.go:275] no items to output this cycle E1121 11:46:46.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:46:46.413264 3798194 memory.go:184] no items to output this cycle I1121 11:46:46.413414 3798194 cpu.go:275] no items to output this cycle E1121 11:46:56.412844 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:46:56.412878 3798194 memory.go:191] Add success. W1121 11:46:56.412906 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:46:56.412917 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:46:56.412922 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:46:56.417081 3798194 cpu.go:282] Add success. I1121 11:46:56.441679 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:46:56.442169 3798194 net.go:642] Add success. I1121 11:46:56.443596 3798194 net.go:778] primary dev: ETH0 I1121 11:46:56.443611 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:46:56.443626 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:46:56.811155 3798194 disk_info.go:125] begin check local disk info of client I1121 11:46:56.815016 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:46:56.815025 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003345c0 0xc000334600] I1121 11:46:57.450894 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:46:57.451084 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:46:57.451095 3798194 disk_worker.go:723] disk space is not compliant W1121 11:46:57.451098 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:46:57.459808 3798194 disk_worker.go:509] system disk:vda1 I1121 11:46:57.459849 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:46:58.446503 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:46:59.450614 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:46:59.450699 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:46:59.450726 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:46:59.471191 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:47:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:47:06.411979 3798194 memory.go:184] no items to output this cycle I1121 11:47:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 11:47:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:47:16.411976 3798194 memory.go:184] no items to output this cycle I1121 11:47:16.412141 3798194 cpu.go:275] no items to output this cycle E1121 11:47:26.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:47:26.411953 3798194 memory.go:191] Add success. I1121 11:47:26.412305 3798194 cpu.go:282] Add success. I1121 11:47:26.441917 3798194 net.go:642] Add success. I1121 11:47:26.446513 3798194 net.go:778] primary dev: ETH0 I1121 11:47:26.446527 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:47:26.446541 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:47:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:47:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:47:29.450144 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:47:36.412819 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:47:36.412848 3798194 memory.go:184] no items to output this cycle I1121 11:47:36.412998 3798194 cpu.go:275] no items to output this cycle E1121 11:47:46.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:47:46.412258 3798194 memory.go:184] no items to output this cycle I1121 11:47:46.412415 3798194 cpu.go:275] no items to output this cycle E1121 11:47:56.411909 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:47:56.411941 3798194 memory.go:191] Add success. W1121 11:47:56.411968 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:47:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:47:56.411982 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:47:56.412328 3798194 cpu.go:282] Add success. I1121 11:47:56.441786 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:47:56.442297 3798194 net.go:642] Add success. I1121 11:47:56.446407 3798194 net.go:778] primary dev: ETH0 I1121 11:47:56.446423 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:47:56.446438 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:47:56.815138 3798194 disk_info.go:125] begin check local disk info of client I1121 11:47:56.819013 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:47:56.819021 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047dc80 0xc00047dcc0] I1121 11:47:57.450545 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:47:57.450754 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:47:57.450765 3798194 disk_worker.go:723] disk space is not compliant W1121 11:47:57.450768 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:47:57.459277 3798194 disk_worker.go:509] system disk:vda1 I1121 11:47:57.459390 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:47:58.446574 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:47:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:47:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:47:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:47:59.471647 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:48:06.411373 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:48:06.411395 3798194 memory.go:184] no items to output this cycle I1121 11:48:06.411536 3798194 cpu.go:275] no items to output this cycle E1121 11:48:16.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:48:16.413253 3798194 memory.go:184] no items to output this cycle I1121 11:48:16.413407 3798194 cpu.go:275] no items to output this cycle I1121 11:48:17.455288 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:48:17.455296 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:48:26.413322 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:48:26.413354 3798194 memory.go:191] Add success. I1121 11:48:26.417651 3798194 cpu.go:282] Add success. I1121 11:48:26.441245 3798194 net.go:642] Add success. I1121 11:48:26.442811 3798194 net.go:778] primary dev: ETH0 I1121 11:48:26.442827 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:48:26.442841 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:48:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:48:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:48:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:48:36.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:48:36.412260 3798194 memory.go:184] no items to output this cycle I1121 11:48:36.412398 3798194 cpu.go:275] no items to output this cycle E1121 11:48:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:48:46.412253 3798194 memory.go:184] no items to output this cycle I1121 11:48:46.412398 3798194 cpu.go:275] no items to output this cycle E1121 11:48:56.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:48:56.412265 3798194 memory.go:191] Add success. W1121 11:48:56.412296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:48:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:48:56.412310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:48:56.416625 3798194 cpu.go:282] Add success. I1121 11:48:56.441135 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:48:56.441624 3798194 net.go:642] Add success. I1121 11:48:56.443205 3798194 net.go:778] primary dev: ETH0 I1121 11:48:56.443221 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:48:56.443233 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:48:56.820143 3798194 disk_info.go:125] begin check local disk info of client I1121 11:48:56.824048 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:48:56.824058 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a4800 0xc0003a4840] I1121 11:48:57.450974 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:48:57.451170 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:48:57.451227 3798194 disk_worker.go:723] disk space is not compliant W1121 11:48:57.451231 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:48:57.460003 3798194 disk_worker.go:509] system disk:vda1 I1121 11:48:57.460044 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:48:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:48:59.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:48:59.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:48:59.450501 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:48:59.471021 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:49:06.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:49:06.413256 3798194 memory.go:184] no items to output this cycle I1121 11:49:06.413397 3798194 cpu.go:275] no items to output this cycle E1121 11:49:16.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:49:16.413254 3798194 memory.go:184] no items to output this cycle I1121 11:49:16.413408 3798194 cpu.go:275] no items to output this cycle E1121 11:49:26.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:49:26.412264 3798194 memory.go:191] Add success. I1121 11:49:26.416690 3798194 cpu.go:282] Add success. I1121 11:49:26.442368 3798194 net.go:642] Add success. I1121 11:49:26.443874 3798194 net.go:778] primary dev: ETH0 I1121 11:49:26.443889 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:49:26.443918 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:49:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:49:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:49:29.450511 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:49:36.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:49:36.412252 3798194 memory.go:184] no items to output this cycle I1121 11:49:36.412400 3798194 cpu.go:275] no items to output this cycle E1121 11:49:46.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:49:46.412260 3798194 memory.go:184] no items to output this cycle I1121 11:49:46.412403 3798194 cpu.go:275] no items to output this cycle E1121 11:49:56.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:49:56.412268 3798194 memory.go:191] Add success. W1121 11:49:56.412301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:49:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:49:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:49:56.416124 3798194 cpu.go:282] Add success. I1121 11:49:56.441424 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:49:56.441903 3798194 net.go:642] Add success. I1121 11:49:56.443377 3798194 net.go:778] primary dev: ETH0 I1121 11:49:56.443393 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:49:56.443407 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:49:56.825148 3798194 disk_info.go:125] begin check local disk info of client I1121 11:49:56.829002 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:49:56.829011 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00048cc00 0xc00048cc40] I1121 11:49:57.450777 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:49:57.451000 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:49:57.451011 3798194 disk_worker.go:723] disk space is not compliant W1121 11:49:57.451014 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:49:57.459687 3798194 disk_worker.go:509] system disk:vda1 I1121 11:49:57.459728 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:49:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:49:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:49:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:49:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:49:59.471699 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:50:06.412865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:50:06.412887 3798194 memory.go:184] no items to output this cycle I1121 11:50:06.413069 3798194 cpu.go:275] no items to output this cycle E1121 11:50:16.412865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:50:16.412887 3798194 memory.go:184] no items to output this cycle I1121 11:50:16.413032 3798194 cpu.go:275] no items to output this cycle E1121 11:50:26.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:50:26.412886 3798194 memory.go:191] Add success. I1121 11:50:26.416940 3798194 cpu.go:282] Add success. I1121 11:50:26.441505 3798194 net.go:642] Add success. I1121 11:50:26.443013 3798194 net.go:778] primary dev: ETH0 I1121 11:50:26.443027 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:50:26.443040 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:50:29.450006 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:50:29.450083 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:50:29.450117 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:50:36.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:50:36.412873 3798194 memory.go:184] no items to output this cycle I1121 11:50:36.413012 3798194 cpu.go:275] no items to output this cycle E1121 11:50:46.412381 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:50:46.412404 3798194 memory.go:184] no items to output this cycle I1121 11:50:46.412553 3798194 cpu.go:275] no items to output this cycle E1121 11:50:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:50:56.411951 3798194 memory.go:191] Add success. W1121 11:50:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:50:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:50:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:50:56.412356 3798194 cpu.go:282] Add success. I1121 11:50:56.444573 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:50:56.445249 3798194 net.go:642] Add success. I1121 11:50:56.449554 3798194 net.go:778] primary dev: ETH0 I1121 11:50:56.449571 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:50:56.449589 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:50:56.829130 3798194 disk_info.go:125] begin check local disk info of client I1121 11:50:56.833077 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:50:56.833087 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004dc080 0xc0004dc0c0] E1121 11:50:57.445590 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 11:50:57.445609 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:50:57.445803 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:50:57.445864 3798194 disk_worker.go:723] disk space is not compliant W1121 11:50:57.445867 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:50:57.450781 3798194 disk_worker.go:509] system disk:vda1 I1121 11:50:57.450822 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 11:50:58.448455 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 11:50:58.448470 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 11:50:59.448976 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 11:50:59.450050 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:50:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:50:59.450126 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:50:59.471654 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:51:06.412767 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:51:06.412789 3798194 memory.go:184] no items to output this cycle I1121 11:51:06.412942 3798194 cpu.go:275] no items to output this cycle E1121 11:51:16.413004 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:51:16.413029 3798194 memory.go:184] no items to output this cycle I1121 11:51:16.413208 3798194 cpu.go:275] no items to output this cycle I1121 11:51:17.456263 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:51:17.456271 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:51:26.411546 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:51:26.411579 3798194 memory.go:191] Add success. I1121 11:51:26.411931 3798194 cpu.go:282] Add success. I1121 11:51:26.441572 3798194 net.go:642] Add success. I1121 11:51:26.448176 3798194 net.go:778] primary dev: ETH0 I1121 11:51:26.448193 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:51:26.448208 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:51:29.450005 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:51:29.450085 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:51:29.450120 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:51:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:51:36.413256 3798194 memory.go:184] no items to output this cycle I1121 11:51:36.413399 3798194 cpu.go:275] no items to output this cycle E1121 11:51:46.411273 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:51:46.411298 3798194 memory.go:184] no items to output this cycle I1121 11:51:46.411449 3798194 cpu.go:275] no items to output this cycle E1121 11:51:56.411877 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:51:56.411911 3798194 memory.go:191] Add success. W1121 11:51:56.411941 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:51:56.411953 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:51:56.411956 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:51:56.416315 3798194 cpu.go:282] Add success. I1121 11:51:56.441919 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:51:56.442453 3798194 net.go:642] Add success. I1121 11:51:56.443857 3798194 net.go:778] primary dev: ETH0 I1121 11:51:56.443874 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:51:56.443886 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:51:56.834140 3798194 disk_info.go:125] begin check local disk info of client I1121 11:51:56.837982 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:51:56.837990 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d0840 0xc0003d0880] I1121 11:51:57.450935 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:51:57.451089 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:51:57.451168 3798194 disk_worker.go:723] disk space is not compliant W1121 11:51:57.451171 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:51:57.460097 3798194 disk_worker.go:509] system disk:vda1 I1121 11:51:57.460157 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:51:58.445663 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:51:59.449678 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:51:59.449758 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:51:59.449786 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:51:59.471302 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:52:06.413262 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:52:06.413285 3798194 memory.go:184] no items to output this cycle I1121 11:52:06.413427 3798194 cpu.go:275] no items to output this cycle E1121 11:52:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:52:16.413266 3798194 memory.go:184] no items to output this cycle I1121 11:52:16.413620 3798194 cpu.go:275] no items to output this cycle E1121 11:52:26.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:52:26.412883 3798194 memory.go:191] Add success. I1121 11:52:26.417189 3798194 cpu.go:282] Add success. I1121 11:52:26.441758 3798194 net.go:642] Add success. I1121 11:52:26.443431 3798194 net.go:778] primary dev: ETH0 I1121 11:52:26.443449 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:52:26.443462 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:52:29.450195 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:52:29.450272 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:52:29.450300 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:52:36.412224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:52:36.412247 3798194 memory.go:184] no items to output this cycle I1121 11:52:36.412401 3798194 cpu.go:275] no items to output this cycle E1121 11:52:46.412773 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:52:46.412805 3798194 memory.go:184] no items to output this cycle I1121 11:52:46.412953 3798194 cpu.go:275] no items to output this cycle E1121 11:52:56.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:52:56.412886 3798194 memory.go:191] Add success. W1121 11:52:56.412919 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:52:56.412931 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:52:56.412934 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:52:56.417462 3798194 cpu.go:282] Add success. I1121 11:52:56.441918 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:52:56.442493 3798194 net.go:642] Add success. I1121 11:52:56.444048 3798194 net.go:778] primary dev: ETH0 I1121 11:52:56.444062 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:52:56.444076 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:52:56.838097 3798194 disk_info.go:125] begin check local disk info of client I1121 11:52:56.846660 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:52:56.846670 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000322340 0xc000322380] I1121 11:52:57.450981 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:52:57.451174 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:52:57.451233 3798194 disk_worker.go:723] disk space is not compliant W1121 11:52:57.451236 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:52:57.460520 3798194 disk_worker.go:509] system disk:vda1 I1121 11:52:57.460562 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:52:58.446014 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:52:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:52:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:52:59.450140 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:52:59.471653 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:53:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:53:06.411983 3798194 memory.go:184] no items to output this cycle I1121 11:53:06.412137 3798194 cpu.go:275] no items to output this cycle E1121 11:53:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:53:16.411985 3798194 memory.go:184] no items to output this cycle I1121 11:53:16.412155 3798194 cpu.go:275] no items to output this cycle E1121 11:53:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:53:26.411952 3798194 memory.go:191] Add success. I1121 11:53:26.412312 3798194 cpu.go:282] Add success. I1121 11:53:26.441958 3798194 net.go:642] Add success. I1121 11:53:26.446383 3798194 net.go:778] primary dev: ETH0 I1121 11:53:26.446406 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:53:26.446420 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:53:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:53:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:53:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:53:36.412962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:53:36.412985 3798194 memory.go:184] no items to output this cycle I1121 11:53:36.413143 3798194 cpu.go:275] no items to output this cycle E1121 11:53:46.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:53:46.412267 3798194 memory.go:184] no items to output this cycle I1121 11:53:46.412413 3798194 cpu.go:275] no items to output this cycle E1121 11:53:56.412287 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:53:56.412325 3798194 memory.go:191] Add success. W1121 11:53:56.412362 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:53:56.412373 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:53:56.412377 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:53:56.416585 3798194 cpu.go:282] Add success. I1121 11:53:56.440921 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:53:56.441465 3798194 net.go:642] Add success. I1121 11:53:56.442958 3798194 net.go:778] primary dev: ETH0 I1121 11:53:56.442973 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:53:56.442985 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:53:56.848223 3798194 disk_info.go:125] begin check local disk info of client I1121 11:53:56.852459 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:53:56.852468 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030fc00 0xc00030fc40] I1121 11:53:57.445917 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:53:57.446135 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:53:57.446208 3798194 disk_worker.go:723] disk space is not compliant W1121 11:53:57.446211 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:53:57.448331 3798194 disk_worker.go:509] system disk:vda1 I1121 11:53:57.448369 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:53:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:53:59.450477 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:53:59.450555 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:53:59.450580 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:53:59.471024 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:54:06.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:54:06.413260 3798194 memory.go:184] no items to output this cycle I1121 11:54:06.413410 3798194 cpu.go:275] no items to output this cycle E1121 11:54:16.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:54:16.413270 3798194 memory.go:184] no items to output this cycle I1121 11:54:16.413436 3798194 cpu.go:275] no items to output this cycle I1121 11:54:17.457144 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:54:17.457153 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:54:26.412889 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:54:26.412923 3798194 memory.go:191] Add success. I1121 11:54:26.417127 3798194 cpu.go:282] Add success. I1121 11:54:26.441647 3798194 net.go:642] Add success. I1121 11:54:26.443273 3798194 net.go:778] primary dev: ETH0 I1121 11:54:26.443288 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:54:26.443300 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:54:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:54:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:54:29.450496 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:54:36.413067 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:54:36.413092 3798194 memory.go:184] no items to output this cycle I1121 11:54:36.413257 3798194 cpu.go:275] no items to output this cycle E1121 11:54:46.413277 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:54:46.413304 3798194 memory.go:184] no items to output this cycle I1121 11:54:46.413507 3798194 cpu.go:275] no items to output this cycle E1121 11:54:56.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:54:56.413268 3798194 memory.go:191] Add success. W1121 11:54:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:54:56.413315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:54:56.413319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:54:56.417686 3798194 cpu.go:282] Add success. I1121 11:54:56.441096 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:54:56.441718 3798194 net.go:642] Add success. I1121 11:54:56.443234 3798194 net.go:778] primary dev: ETH0 I1121 11:54:56.443251 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:54:56.443264 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:54:56.853137 3798194 disk_info.go:125] begin check local disk info of client I1121 11:54:56.859207 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:54:56.859217 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000446780 0xc0004467c0] I1121 11:54:57.450840 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:54:57.451063 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:54:57.451075 3798194 disk_worker.go:723] disk space is not compliant W1121 11:54:57.451078 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:54:57.461747 3798194 disk_worker.go:509] system disk:vda1 I1121 11:54:57.461791 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:54:58.446218 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:54:59.450222 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:54:59.450317 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:54:59.450346 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:54:59.470879 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:55:06.412065 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:55:06.412179 3798194 memory.go:184] no items to output this cycle I1121 11:55:06.412603 3798194 cpu.go:275] no items to output this cycle E1121 11:55:16.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:55:16.413248 3798194 memory.go:184] no items to output this cycle I1121 11:55:16.413400 3798194 cpu.go:275] no items to output this cycle E1121 11:55:26.413223 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:55:26.413257 3798194 memory.go:191] Add success. I1121 11:55:26.417574 3798194 cpu.go:282] Add success. I1121 11:55:26.442145 3798194 net.go:642] Add success. I1121 11:55:26.443581 3798194 net.go:778] primary dev: ETH0 I1121 11:55:26.443596 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:55:26.443610 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:55:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:55:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:55:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:55:36.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:55:36.412876 3798194 memory.go:184] no items to output this cycle I1121 11:55:36.413018 3798194 cpu.go:275] no items to output this cycle E1121 11:55:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:55:46.411987 3798194 memory.go:184] no items to output this cycle I1121 11:55:46.412167 3798194 cpu.go:275] no items to output this cycle E1121 11:55:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:55:56.411948 3798194 memory.go:191] Add success. W1121 11:55:56.411975 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:55:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:55:56.411990 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:55:56.412320 3798194 cpu.go:282] Add success. I1121 11:55:56.441521 3798194 event_worker.go:152] Polling the log file for events... I1121 11:55:56.444652 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:55:56.445147 3798194 net.go:642] Add success. I1121 11:55:56.449317 3798194 net.go:778] primary dev: ETH0 I1121 11:55:56.449331 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:55:56.449345 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:55:56.860148 3798194 disk_info.go:125] begin check local disk info of client I1121 11:55:56.863995 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:55:56.864004 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000353a00 0xc000353a40] E1121 11:55:57.450530 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 11:55:57.450544 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:55:57.450736 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:55:57.450746 3798194 disk_worker.go:723] disk space is not compliant W1121 11:55:57.450749 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:55:57.459306 3798194 disk_worker.go:509] system disk:vda1 I1121 11:55:57.459349 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 11:55:58.448089 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 11:55:58.448105 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 11:55:59.448606 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 11:55:59.449681 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:55:59.449734 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:55:59.449752 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:55:59.471189 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:56:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:56:06.412254 3798194 memory.go:184] no items to output this cycle I1121 11:56:06.412399 3798194 cpu.go:275] no items to output this cycle E1121 11:56:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:56:16.413257 3798194 memory.go:184] no items to output this cycle I1121 11:56:16.413414 3798194 cpu.go:275] no items to output this cycle E1121 11:56:26.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:56:26.412260 3798194 memory.go:191] Add success. I1121 11:56:26.416709 3798194 cpu.go:282] Add success. I1121 11:56:26.442263 3798194 net.go:642] Add success. I1121 11:56:26.443876 3798194 net.go:778] primary dev: ETH0 I1121 11:56:26.443893 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:56:26.443912 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:56:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:56:29.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:56:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:56:36.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:56:36.412885 3798194 memory.go:184] no items to output this cycle I1121 11:56:36.413027 3798194 cpu.go:275] no items to output this cycle E1121 11:56:46.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:56:46.411991 3798194 memory.go:184] no items to output this cycle I1121 11:56:46.412165 3798194 cpu.go:275] no items to output this cycle E1121 11:56:56.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:56:56.411961 3798194 memory.go:191] Add success. W1121 11:56:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:56:56.412008 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:56:56.412011 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:56:56.412348 3798194 cpu.go:282] Add success. I1121 11:56:56.441922 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:56:56.442493 3798194 net.go:642] Add success. I1121 11:56:56.446825 3798194 net.go:778] primary dev: ETH0 I1121 11:56:56.446840 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:56:56.446852 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:56:56.865243 3798194 disk_info.go:125] begin check local disk info of client I1121 11:56:56.871368 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:56:56.871377 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d8140 0xc0003d8180] I1121 11:56:57.450552 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:56:57.450724 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:56:57.450778 3798194 disk_worker.go:723] disk space is not compliant W1121 11:56:57.450781 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:56:57.461649 3798194 disk_worker.go:509] system disk:vda1 I1121 11:56:57.461693 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:56:58.445995 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:56:59.450091 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:56:59.450179 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:56:59.450207 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:56:59.470670 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:57:06.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:57:06.411975 3798194 memory.go:184] no items to output this cycle I1121 11:57:06.412129 3798194 cpu.go:275] no items to output this cycle E1121 11:57:16.412359 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:57:16.412381 3798194 memory.go:184] no items to output this cycle I1121 11:57:16.412529 3798194 cpu.go:275] no items to output this cycle I1121 11:57:17.457308 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 11:57:17.457316 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 11:57:26.411361 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:57:26.411398 3798194 memory.go:191] Add success. I1121 11:57:26.411715 3798194 cpu.go:282] Add success. I1121 11:57:26.441515 3798194 net.go:642] Add success. I1121 11:57:26.445367 3798194 net.go:778] primary dev: ETH0 I1121 11:57:26.445383 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:57:26.445398 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:57:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:57:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:57:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:57:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:57:36.412256 3798194 memory.go:184] no items to output this cycle I1121 11:57:36.412402 3798194 cpu.go:275] no items to output this cycle E1121 11:57:46.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:57:46.412874 3798194 memory.go:184] no items to output this cycle I1121 11:57:46.413025 3798194 cpu.go:275] no items to output this cycle E1121 11:57:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:57:56.411949 3798194 memory.go:191] Add success. W1121 11:57:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:57:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:57:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:57:56.412361 3798194 cpu.go:282] Add success. I1121 11:57:56.442027 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:57:56.443005 3798194 net.go:642] Add success. I1121 11:57:56.446849 3798194 net.go:778] primary dev: ETH0 I1121 11:57:56.446864 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:57:56.446882 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:57:56.871485 3798194 disk_info.go:125] begin check local disk info of client I1121 11:57:56.875758 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:57:56.875767 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00049e080 0xc00049e0c0] I1121 11:57:57.450681 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:57:57.450890 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:57:57.450900 3798194 disk_worker.go:723] disk space is not compliant W1121 11:57:57.450903 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:57:57.463884 3798194 disk_worker.go:509] system disk:vda1 I1121 11:57:57.463930 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:57:58.445988 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:57:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:57:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:57:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:57:59.471660 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:58:06.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:58:06.411973 3798194 memory.go:184] no items to output this cycle I1121 11:58:06.412130 3798194 cpu.go:275] no items to output this cycle E1121 11:58:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:58:16.412258 3798194 memory.go:184] no items to output this cycle I1121 11:58:16.412410 3798194 cpu.go:275] no items to output this cycle E1121 11:58:26.412791 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:58:26.412824 3798194 memory.go:191] Add success. I1121 11:58:26.416767 3798194 cpu.go:282] Add success. I1121 11:58:26.441431 3798194 net.go:642] Add success. I1121 11:58:26.442983 3798194 net.go:778] primary dev: ETH0 I1121 11:58:26.442999 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:58:26.443012 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:58:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:58:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:58:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:58:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:58:36.413257 3798194 memory.go:184] no items to output this cycle I1121 11:58:36.413397 3798194 cpu.go:275] no items to output this cycle E1121 11:58:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:58:46.412259 3798194 memory.go:184] no items to output this cycle I1121 11:58:46.412417 3798194 cpu.go:275] no items to output this cycle E1121 11:58:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:58:56.411947 3798194 memory.go:191] Add success. W1121 11:58:56.411975 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:58:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:58:56.411989 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:58:56.412341 3798194 cpu.go:282] Add success. I1121 11:58:56.441755 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:58:56.442256 3798194 net.go:642] Add success. I1121 11:58:56.446292 3798194 net.go:778] primary dev: ETH0 I1121 11:58:56.446306 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:58:56.446319 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:58:56.876134 3798194 disk_info.go:125] begin check local disk info of client I1121 11:58:56.880004 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:58:56.880013 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d6fc0 0xc0003d7000] I1121 11:58:57.450573 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:58:57.450786 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:58:57.450798 3798194 disk_worker.go:723] disk space is not compliant W1121 11:58:57.450801 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:58:57.459192 3798194 disk_worker.go:509] system disk:vda1 I1121 11:58:57.459235 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:58:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:58:59.450101 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:58:59.450185 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:58:59.450211 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:58:59.470686 3798194 disk_local_worker.go:436] Get disk info: [] E1121 11:59:06.413162 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:59:06.413183 3798194 memory.go:184] no items to output this cycle I1121 11:59:06.413549 3798194 cpu.go:275] no items to output this cycle E1121 11:59:16.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:59:16.413259 3798194 memory.go:184] no items to output this cycle I1121 11:59:16.413409 3798194 cpu.go:275] no items to output this cycle E1121 11:59:26.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:59:26.412903 3798194 memory.go:191] Add success. I1121 11:59:26.417767 3798194 cpu.go:282] Add success. I1121 11:59:26.441281 3798194 net.go:642] Add success. I1121 11:59:26.443010 3798194 net.go:778] primary dev: ETH0 I1121 11:59:26.443025 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:59:26.443038 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:59:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:59:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:59:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 11:59:36.411051 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:59:36.411073 3798194 memory.go:184] no items to output this cycle I1121 11:59:36.411242 3798194 cpu.go:275] no items to output this cycle E1121 11:59:46.412846 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:59:46.412867 3798194 memory.go:184] no items to output this cycle I1121 11:59:46.413023 3798194 cpu.go:275] no items to output this cycle E1121 11:59:56.412797 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 11:59:56.412831 3798194 memory.go:191] Add success. W1121 11:59:56.412856 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 11:59:56.412868 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 11:59:56.412871 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 11:59:56.417622 3798194 cpu.go:282] Add success. I1121 11:59:56.440947 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 11:59:56.441516 3798194 net.go:642] Add success. I1121 11:59:56.443045 3798194 net.go:778] primary dev: ETH0 I1121 11:59:56.443060 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 11:59:56.443075 3798194 net.go:659] Add eni metrics success, count: 7 I1121 11:59:56.881152 3798194 disk_info.go:125] begin check local disk info of client I1121 11:59:56.885153 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 11:59:56.885163 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000361640 0xc000361680] I1121 11:59:57.445876 3798194 custom_config.go:106] query custom config with name: gpu W1121 11:59:57.446151 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 11:59:57.446163 3798194 disk_worker.go:723] disk space is not compliant W1121 11:59:57.446166 3798194 disk_worker.go:743] disk inode is not compliant I1121 11:59:57.448329 3798194 disk_worker.go:509] system disk:vda1 I1121 11:59:57.448368 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 11:59:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 11:59:59.450479 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 11:59:59.450554 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 11:59:59.450579 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 11:59:59.471134 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:00:06.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:00:06.412253 3798194 memory.go:184] no items to output this cycle I1121 12:00:06.412402 3798194 cpu.go:275] no items to output this cycle E1121 12:00:16.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:00:16.412259 3798194 memory.go:184] no items to output this cycle I1121 12:00:16.412407 3798194 cpu.go:275] no items to output this cycle I1121 12:00:17.457470 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:00:17.457479 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:00:26.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:00:26.412270 3798194 memory.go:191] Add success. I1121 12:00:26.416346 3798194 cpu.go:282] Add success. I1121 12:00:26.441873 3798194 net.go:642] Add success. I1121 12:00:26.443245 3798194 net.go:778] primary dev: ETH0 I1121 12:00:26.443261 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:00:26.443274 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:00:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:00:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:00:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:00:36.412339 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:00:36.412377 3798194 memory.go:184] no items to output this cycle I1121 12:00:36.412643 3798194 cpu.go:275] no items to output this cycle E1121 12:00:46.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:00:46.412251 3798194 memory.go:184] no items to output this cycle I1121 12:00:46.412397 3798194 cpu.go:275] no items to output this cycle E1121 12:00:56.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:00:56.412273 3798194 memory.go:191] Add success. W1121 12:00:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:00:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:00:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:00:56.412675 3798194 cpu.go:282] Add success. I1121 12:00:56.444038 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:00:56.444546 3798194 net.go:642] Add success. I1121 12:00:56.448491 3798194 net.go:778] primary dev: ETH0 I1121 12:00:56.448506 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:00:56.448519 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:00:56.886146 3798194 disk_info.go:125] begin check local disk info of client I1121 12:00:56.889903 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:00:56.889911 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000360a80 0xc000360ac0] E1121 12:00:57.450995 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 12:00:57.451009 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:00:57.451222 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:00:57.451231 3798194 disk_worker.go:723] disk space is not compliant W1121 12:00:57.451235 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:00:57.460077 3798194 disk_worker.go:509] system disk:vda1 I1121 12:00:57.460141 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 12:00:58.448465 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 12:00:58.448478 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 12:00:59.449160 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 12:00:59.450234 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:00:59.450286 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:00:59.450305 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:00:59.470731 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:01:06.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:01:06.413261 3798194 memory.go:184] no items to output this cycle I1121 12:01:06.413405 3798194 cpu.go:275] no items to output this cycle E1121 12:01:16.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:01:16.411990 3798194 memory.go:184] no items to output this cycle I1121 12:01:16.412154 3798194 cpu.go:275] no items to output this cycle E1121 12:01:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:01:26.411957 3798194 memory.go:191] Add success. I1121 12:01:26.412317 3798194 cpu.go:282] Add success. I1121 12:01:26.442030 3798194 net.go:642] Add success. I1121 12:01:26.446072 3798194 net.go:778] primary dev: ETH0 I1121 12:01:26.446087 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:01:26.446100 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:01:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:01:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:01:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:01:36.412810 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:01:36.412833 3798194 memory.go:184] no items to output this cycle I1121 12:01:36.412984 3798194 cpu.go:275] no items to output this cycle E1121 12:01:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:01:46.413263 3798194 memory.go:184] no items to output this cycle I1121 12:01:46.413406 3798194 cpu.go:275] no items to output this cycle E1121 12:01:56.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:01:56.412893 3798194 memory.go:191] Add success. W1121 12:01:56.412922 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:01:56.412933 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:01:56.412937 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:01:56.417376 3798194 cpu.go:282] Add success. I1121 12:01:56.441870 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:01:56.442492 3798194 net.go:642] Add success. I1121 12:01:56.444176 3798194 net.go:778] primary dev: ETH0 I1121 12:01:56.444192 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:01:56.444209 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:01:56.890090 3798194 disk_info.go:125] begin check local disk info of client I1121 12:01:56.893929 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:01:56.893937 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000491080 0xc0004910c0] I1121 12:01:57.450696 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:01:57.450897 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:01:57.450907 3798194 disk_worker.go:723] disk space is not compliant W1121 12:01:57.450910 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:01:57.459205 3798194 disk_worker.go:509] system disk:vda1 I1121 12:01:57.459248 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:01:58.445994 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:01:59.450094 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:01:59.450182 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:01:59.450207 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:01:59.470691 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:02:06.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:02:06.413251 3798194 memory.go:184] no items to output this cycle I1121 12:02:06.413396 3798194 cpu.go:275] no items to output this cycle E1121 12:02:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:02:16.413259 3798194 memory.go:184] no items to output this cycle I1121 12:02:16.413418 3798194 cpu.go:275] no items to output this cycle E1121 12:02:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:02:26.411953 3798194 memory.go:191] Add success. I1121 12:02:26.412311 3798194 cpu.go:282] Add success. I1121 12:02:26.442040 3798194 net.go:642] Add success. I1121 12:02:26.446322 3798194 net.go:778] primary dev: ETH0 I1121 12:02:26.446341 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:02:26.446355 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:02:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:02:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:02:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:02:36.411944 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:02:36.411966 3798194 memory.go:184] no items to output this cycle I1121 12:02:36.412105 3798194 cpu.go:275] no items to output this cycle E1121 12:02:46.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:02:46.412878 3798194 memory.go:184] no items to output this cycle I1121 12:02:46.413025 3798194 cpu.go:275] no items to output this cycle E1121 12:02:56.413045 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:02:56.413082 3798194 memory.go:191] Add success. W1121 12:02:56.413133 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:02:56.413146 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:02:56.413150 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:02:56.416985 3798194 cpu.go:282] Add success. I1121 12:02:56.441423 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:02:56.441888 3798194 net.go:642] Add success. I1121 12:02:56.443396 3798194 net.go:778] primary dev: ETH0 I1121 12:02:56.443412 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:02:56.443424 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:02:56.895136 3798194 disk_info.go:125] begin check local disk info of client I1121 12:02:56.899070 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:02:56.899079 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ca680 0xc0003ca6c0] I1121 12:02:57.450312 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:02:57.450474 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:02:57.450530 3798194 disk_worker.go:723] disk space is not compliant W1121 12:02:57.450533 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:02:57.458862 3798194 disk_worker.go:509] system disk:vda1 I1121 12:02:57.458904 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:02:58.446029 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:02:59.450132 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:02:59.450215 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:02:59.450240 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:02:59.470713 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:03:06.412098 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:03:06.412139 3798194 memory.go:184] no items to output this cycle I1121 12:03:06.412304 3798194 cpu.go:275] no items to output this cycle E1121 12:03:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:03:16.411985 3798194 memory.go:184] no items to output this cycle I1121 12:03:16.412145 3798194 cpu.go:275] no items to output this cycle I1121 12:03:17.458493 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:03:17.458503 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:03:26.411344 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:03:26.411379 3798194 memory.go:191] Add success. I1121 12:03:26.411735 3798194 cpu.go:282] Add success. I1121 12:03:26.441632 3798194 net.go:642] Add success. I1121 12:03:26.451492 3798194 net.go:778] primary dev: ETH0 I1121 12:03:26.451509 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:03:26.451523 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:03:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:03:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:03:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:03:36.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:03:36.413253 3798194 memory.go:184] no items to output this cycle I1121 12:03:36.413395 3798194 cpu.go:275] no items to output this cycle E1121 12:03:46.412843 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:03:46.412867 3798194 memory.go:184] no items to output this cycle I1121 12:03:46.413011 3798194 cpu.go:275] no items to output this cycle E1121 12:03:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:03:56.412268 3798194 memory.go:191] Add success. W1121 12:03:56.412298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:03:56.412310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:03:56.412313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:03:56.416759 3798194 cpu.go:282] Add success. I1121 12:03:56.441428 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:03:56.442004 3798194 net.go:642] Add success. I1121 12:03:56.443451 3798194 net.go:778] primary dev: ETH0 I1121 12:03:56.443467 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:03:56.443480 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:03:56.901145 3798194 disk_info.go:125] begin check local disk info of client I1121 12:03:56.905104 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:03:56.905131 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003127c0 0xc000312800] I1121 12:03:57.450621 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:03:57.450826 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:03:57.450838 3798194 disk_worker.go:723] disk space is not compliant W1121 12:03:57.450841 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:03:57.459216 3798194 disk_worker.go:509] system disk:vda1 I1121 12:03:57.459258 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:03:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:03:59.450606 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:03:59.450675 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:03:59.450700 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:03:59.471185 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:04:06.413426 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:04:06.413449 3798194 memory.go:184] no items to output this cycle I1121 12:04:06.413590 3798194 cpu.go:275] no items to output this cycle E1121 12:04:16.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:04:16.411988 3798194 memory.go:184] no items to output this cycle I1121 12:04:16.412152 3798194 cpu.go:275] no items to output this cycle E1121 12:04:26.411930 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:04:26.411964 3798194 memory.go:191] Add success. I1121 12:04:26.412323 3798194 cpu.go:282] Add success. I1121 12:04:26.442082 3798194 net.go:642] Add success. I1121 12:04:26.446801 3798194 net.go:778] primary dev: ETH0 I1121 12:04:26.446818 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:04:26.446831 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:04:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:04:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:04:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:04:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:04:36.413261 3798194 memory.go:184] no items to output this cycle I1121 12:04:36.413411 3798194 cpu.go:275] no items to output this cycle E1121 12:04:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:04:46.412260 3798194 memory.go:184] no items to output this cycle I1121 12:04:46.412407 3798194 cpu.go:275] no items to output this cycle E1121 12:04:56.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:04:56.412262 3798194 memory.go:191] Add success. W1121 12:04:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:04:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:04:56.412311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:04:56.416759 3798194 cpu.go:282] Add success. I1121 12:04:56.441155 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:04:56.441619 3798194 net.go:642] Add success. I1121 12:04:56.443095 3798194 net.go:778] primary dev: ETH0 I1121 12:04:56.443121 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:04:56.443137 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:04:56.906144 3798194 disk_info.go:125] begin check local disk info of client I1121 12:04:56.910059 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:04:56.910068 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036e900 0xc00036e940] I1121 12:04:57.450373 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:04:57.450538 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:04:57.450594 3798194 disk_worker.go:723] disk space is not compliant W1121 12:04:57.450597 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:04:57.459074 3798194 disk_worker.go:509] system disk:vda1 I1121 12:04:57.459131 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:04:58.445884 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:04:59.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:04:59.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:04:59.450498 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:04:59.471006 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:05:06.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:05:06.413261 3798194 memory.go:184] no items to output this cycle I1121 12:05:06.413417 3798194 cpu.go:275] no items to output this cycle E1121 12:05:16.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:05:16.411985 3798194 memory.go:184] no items to output this cycle I1121 12:05:16.412145 3798194 cpu.go:275] no items to output this cycle E1121 12:05:26.411776 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:05:26.411811 3798194 memory.go:191] Add success. I1121 12:05:26.419938 3798194 cpu.go:282] Add success. I1121 12:05:26.441527 3798194 net.go:642] Add success. I1121 12:05:26.443033 3798194 net.go:778] primary dev: ETH0 I1121 12:05:26.443048 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:05:26.443063 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:05:29.450004 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:05:29.450085 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:05:29.450119 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:05:36.411427 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:05:36.411448 3798194 memory.go:184] no items to output this cycle I1121 12:05:36.411599 3798194 cpu.go:275] no items to output this cycle E1121 12:05:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:05:46.411981 3798194 memory.go:184] no items to output this cycle I1121 12:05:46.412143 3798194 cpu.go:275] no items to output this cycle E1121 12:05:56.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:05:56.412264 3798194 memory.go:191] Add success. W1121 12:05:56.412294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:05:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:05:56.412317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:05:56.417577 3798194 cpu.go:282] Add success. I1121 12:05:56.443923 3798194 event_worker.go:152] Polling the log file for events... I1121 12:05:56.444280 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:05:56.444764 3798194 net.go:642] Add success. I1121 12:05:56.446179 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795232156, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 12:05:56.446302 3798194 net.go:778] primary dev: ETH0 I1121 12:05:56.446327 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:05:56.446339 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:05:56.454068 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 I1121 12:05:56.912144 3798194 disk_info.go:125] begin check local disk info of client I1121 12:05:56.916962 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:05:56.916971 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000498100 0xc000498140] E1121 12:05:57.445863 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 12:05:57.445878 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:05:57.446149 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:05:57.446162 3798194 disk_worker.go:723] disk space is not compliant W1121 12:05:57.446165 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:05:57.454454 3798194 disk_worker.go:509] system disk:vda1 I1121 12:05:57.454506 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 12:05:58.448495 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 12:05:58.448510 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 12:05:59.448976 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 12:05:59.450051 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:05:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:05:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:05:59.471651 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:06:06.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:06:06.411985 3798194 memory.go:184] no items to output this cycle I1121 12:06:06.412143 3798194 cpu.go:275] no items to output this cycle E1121 12:06:16.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:06:16.411990 3798194 memory.go:184] no items to output this cycle I1121 12:06:16.412153 3798194 cpu.go:275] no items to output this cycle I1121 12:06:17.459512 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:06:17.459521 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:06:26.412841 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:06:26.412874 3798194 memory.go:191] Add success. I1121 12:06:26.417299 3798194 cpu.go:282] Add success. I1121 12:06:26.441675 3798194 net.go:642] Add success. I1121 12:06:26.443481 3798194 net.go:778] primary dev: ETH0 I1121 12:06:26.443496 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:06:26.443510 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:06:29.450191 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:06:29.450269 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:06:29.450294 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:06:36.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:06:36.411988 3798194 memory.go:184] no items to output this cycle I1121 12:06:36.412143 3798194 cpu.go:275] no items to output this cycle E1121 12:06:46.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:06:46.411968 3798194 memory.go:184] no items to output this cycle I1121 12:06:46.412129 3798194 cpu.go:275] no items to output this cycle E1121 12:06:56.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:06:56.412259 3798194 memory.go:191] Add success. W1121 12:06:56.412291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:06:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:06:56.412306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:06:56.416870 3798194 cpu.go:282] Add success. I1121 12:06:56.441285 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:06:56.441765 3798194 net.go:642] Add success. I1121 12:06:56.443277 3798194 net.go:778] primary dev: ETH0 I1121 12:06:56.443292 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:06:56.443305 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:06:56.919335 3798194 disk_info.go:125] begin check local disk info of client I1121 12:06:56.923562 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:06:56.923571 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00039b340 0xc00039b380] I1121 12:06:57.451040 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:06:57.451228 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:06:57.451283 3798194 disk_worker.go:723] disk space is not compliant W1121 12:06:57.451286 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:06:57.465354 3798194 disk_worker.go:509] system disk:vda1 I1121 12:06:57.465401 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:06:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:06:59.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:06:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:06:59.450502 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:06:59.471021 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:07:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:07:06.412260 3798194 memory.go:184] no items to output this cycle I1121 12:07:06.412403 3798194 cpu.go:275] no items to output this cycle E1121 12:07:16.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:07:16.412265 3798194 memory.go:184] no items to output this cycle I1121 12:07:16.412413 3798194 cpu.go:275] no items to output this cycle E1121 12:07:26.412030 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:07:26.412075 3798194 memory.go:191] Add success. I1121 12:07:26.412716 3798194 cpu.go:282] Add success. I1121 12:07:26.441304 3798194 net.go:642] Add success. I1121 12:07:26.445737 3798194 net.go:778] primary dev: ETH0 I1121 12:07:26.445752 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:07:26.445766 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:07:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:07:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:07:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:07:36.412844 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:07:36.412866 3798194 memory.go:184] no items to output this cycle I1121 12:07:36.413018 3798194 cpu.go:275] no items to output this cycle E1121 12:07:46.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:07:46.412253 3798194 memory.go:184] no items to output this cycle I1121 12:07:46.412395 3798194 cpu.go:275] no items to output this cycle E1121 12:07:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:07:56.413271 3798194 memory.go:191] Add success. W1121 12:07:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:07:56.413315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:07:56.413318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:07:56.417487 3798194 cpu.go:282] Add success. I1121 12:07:56.441118 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:07:56.441604 3798194 net.go:642] Add success. I1121 12:07:56.443296 3798194 net.go:778] primary dev: ETH0 I1121 12:07:56.443312 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:07:56.443324 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:07:56.925156 3798194 disk_info.go:125] begin check local disk info of client I1121 12:07:56.929439 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:07:56.929449 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00039ad80 0xc00039adc0] I1121 12:07:57.451012 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:07:57.451208 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:07:57.451267 3798194 disk_worker.go:723] disk space is not compliant W1121 12:07:57.451270 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:07:57.460076 3798194 disk_worker.go:509] system disk:vda1 I1121 12:07:57.460151 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:07:58.446384 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:07:59.450406 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:07:59.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:07:59.450510 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:07:59.471025 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:08:06.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:08:06.413251 3798194 memory.go:184] no items to output this cycle I1121 12:08:06.413393 3798194 cpu.go:275] no items to output this cycle E1121 12:08:16.413265 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:08:16.413287 3798194 memory.go:184] no items to output this cycle I1121 12:08:16.413444 3798194 cpu.go:275] no items to output this cycle E1121 12:08:26.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:08:26.411950 3798194 memory.go:191] Add success. I1121 12:08:26.412299 3798194 cpu.go:282] Add success. I1121 12:08:26.442031 3798194 net.go:642] Add success. I1121 12:08:26.445825 3798194 net.go:778] primary dev: ETH0 I1121 12:08:26.445840 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:08:26.445855 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:08:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:08:29.450118 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:08:29.450147 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:08:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:08:36.413258 3798194 memory.go:184] no items to output this cycle I1121 12:08:36.413398 3798194 cpu.go:275] no items to output this cycle E1121 12:08:46.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:08:46.413268 3798194 memory.go:184] no items to output this cycle I1121 12:08:46.413534 3798194 cpu.go:275] no items to output this cycle E1121 12:08:56.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:08:56.412272 3798194 memory.go:191] Add success. W1121 12:08:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:08:56.412317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:08:56.412320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:08:56.416541 3798194 cpu.go:282] Add success. I1121 12:08:56.440952 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:08:56.441541 3798194 net.go:642] Add success. I1121 12:08:56.443000 3798194 net.go:778] primary dev: ETH0 I1121 12:08:56.443014 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:08:56.443027 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:08:56.930147 3798194 disk_info.go:125] begin check local disk info of client I1121 12:08:56.934019 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:08:56.934028 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0cc0 0xc0002e0d00] I1121 12:08:57.450556 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:08:57.450762 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:08:57.450773 3798194 disk_worker.go:723] disk space is not compliant W1121 12:08:57.450776 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:08:57.459184 3798194 disk_worker.go:509] system disk:vda1 I1121 12:08:57.459225 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:08:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:08:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:08:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:08:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:08:59.471670 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:09:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:09:06.411983 3798194 memory.go:184] no items to output this cycle I1121 12:09:06.412144 3798194 cpu.go:275] no items to output this cycle E1121 12:09:16.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:09:16.413267 3798194 memory.go:184] no items to output this cycle I1121 12:09:16.413412 3798194 cpu.go:275] no items to output this cycle I1121 12:09:17.459669 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:09:17.459677 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:09:26.411407 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:09:26.411442 3798194 memory.go:191] Add success. I1121 12:09:26.411780 3798194 cpu.go:282] Add success. I1121 12:09:26.441535 3798194 net.go:642] Add success. I1121 12:09:26.445455 3798194 net.go:778] primary dev: ETH0 I1121 12:09:26.445470 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:09:26.445482 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:09:29.450008 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:09:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:09:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:09:36.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:09:36.412263 3798194 memory.go:184] no items to output this cycle I1121 12:09:36.412410 3798194 cpu.go:275] no items to output this cycle E1121 12:09:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:09:46.413256 3798194 memory.go:184] no items to output this cycle I1121 12:09:46.413399 3798194 cpu.go:275] no items to output this cycle E1121 12:09:56.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:09:56.413275 3798194 memory.go:191] Add success. W1121 12:09:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:09:56.413316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:09:56.413320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:09:56.417662 3798194 cpu.go:282] Add success. I1121 12:09:56.441182 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:09:56.441672 3798194 net.go:642] Add success. I1121 12:09:56.443249 3798194 net.go:778] primary dev: ETH0 I1121 12:09:56.443264 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:09:56.443277 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:09:56.936138 3798194 disk_info.go:125] begin check local disk info of client I1121 12:09:56.939921 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:09:56.939930 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000433800 0xc000433840] I1121 12:09:57.450954 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:09:57.451139 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:09:57.451193 3798194 disk_worker.go:723] disk space is not compliant W1121 12:09:57.451196 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:09:57.460348 3798194 disk_worker.go:509] system disk:vda1 I1121 12:09:57.460389 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:09:58.446384 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:09:59.450490 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:09:59.450568 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:09:59.450596 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:09:59.471132 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:10:06.412865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:10:06.412890 3798194 memory.go:184] no items to output this cycle I1121 12:10:06.413040 3798194 cpu.go:275] no items to output this cycle E1121 12:10:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:10:16.411984 3798194 memory.go:184] no items to output this cycle I1121 12:10:16.412146 3798194 cpu.go:275] no items to output this cycle E1121 12:10:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:10:26.411953 3798194 memory.go:191] Add success. I1121 12:10:26.412331 3798194 cpu.go:282] Add success. I1121 12:10:26.442055 3798194 net.go:642] Add success. I1121 12:10:26.446048 3798194 net.go:778] primary dev: ETH0 I1121 12:10:26.446063 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:10:26.446075 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:10:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:10:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:10:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:10:36.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:10:36.413268 3798194 memory.go:184] no items to output this cycle I1121 12:10:36.413412 3798194 cpu.go:275] no items to output this cycle E1121 12:10:46.412951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:10:46.412972 3798194 memory.go:184] no items to output this cycle I1121 12:10:46.413143 3798194 cpu.go:275] no items to output this cycle E1121 12:10:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:10:56.412263 3798194 memory.go:191] Add success. W1121 12:10:56.412294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:10:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:10:56.412309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:10:56.416999 3798194 cpu.go:282] Add success. I1121 12:10:56.444206 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:10:56.444697 3798194 net.go:642] Add success. I1121 12:10:56.446561 3798194 net.go:778] primary dev: ETH0 I1121 12:10:56.446576 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:10:56.446588 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:10:56.941148 3798194 disk_info.go:125] begin check local disk info of client I1121 12:10:56.945034 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:10:56.945042 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c3200 0xc0003c3240] E1121 12:10:57.450935 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 12:10:57.450949 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:10:57.451183 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:10:57.451194 3798194 disk_worker.go:723] disk space is not compliant W1121 12:10:57.451197 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:10:57.459973 3798194 disk_worker.go:509] system disk:vda1 I1121 12:10:57.460016 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 12:10:58.448714 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 12:10:58.448729 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 12:10:59.448265 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 12:10:59.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:10:59.450449 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:10:59.450468 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:10:59.470980 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:11:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:11:06.412259 3798194 memory.go:184] no items to output this cycle I1121 12:11:06.412401 3798194 cpu.go:275] no items to output this cycle E1121 12:11:16.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:11:16.412266 3798194 memory.go:184] no items to output this cycle I1121 12:11:16.412420 3798194 cpu.go:275] no items to output this cycle E1121 12:11:26.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:11:26.412896 3798194 memory.go:191] Add success. I1121 12:11:26.424497 3798194 cpu.go:282] Add success. I1121 12:11:26.442035 3798194 net.go:642] Add success. I1121 12:11:26.443383 3798194 net.go:778] primary dev: ETH0 I1121 12:11:26.443397 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:11:26.443411 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:11:29.450131 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:11:29.450214 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:11:29.450243 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:11:36.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:11:36.413251 3798194 memory.go:184] no items to output this cycle I1121 12:11:36.413394 3798194 cpu.go:275] no items to output this cycle E1121 12:11:46.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:11:46.413266 3798194 memory.go:184] no items to output this cycle I1121 12:11:46.413412 3798194 cpu.go:275] no items to output this cycle E1121 12:11:56.412752 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:11:56.412785 3798194 memory.go:191] Add success. W1121 12:11:56.412816 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:11:56.412828 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:11:56.412831 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:11:56.417186 3798194 cpu.go:282] Add success. I1121 12:11:56.441759 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:11:56.442369 3798194 net.go:642] Add success. I1121 12:11:56.443821 3798194 net.go:778] primary dev: ETH0 I1121 12:11:56.443835 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:11:56.443848 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:11:56.946162 3798194 disk_info.go:125] begin check local disk info of client I1121 12:11:56.950022 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:11:56.950030 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000251ac0 0xc000251b00] I1121 12:11:57.451046 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:11:57.451240 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:11:57.451292 3798194 disk_worker.go:723] disk space is not compliant W1121 12:11:57.451295 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:11:57.460039 3798194 disk_worker.go:509] system disk:vda1 I1121 12:11:57.460078 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:11:58.446021 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:11:59.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:11:59.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:11:59.450508 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:11:59.471032 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:12:06.412847 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:12:06.412872 3798194 memory.go:184] no items to output this cycle I1121 12:12:06.413010 3798194 cpu.go:275] no items to output this cycle E1121 12:12:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:12:16.411983 3798194 memory.go:184] no items to output this cycle I1121 12:12:16.412143 3798194 cpu.go:275] no items to output this cycle I1121 12:12:17.460147 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:12:17.460155 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:12:26.411497 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:12:26.411528 3798194 memory.go:191] Add success. I1121 12:12:26.411865 3798194 cpu.go:282] Add success. I1121 12:12:26.441688 3798194 net.go:642] Add success. I1121 12:12:26.445821 3798194 net.go:778] primary dev: ETH0 I1121 12:12:26.445836 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:12:26.445853 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:12:29.450004 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:12:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:12:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:12:36.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:12:36.412253 3798194 memory.go:184] no items to output this cycle I1121 12:12:36.412392 3798194 cpu.go:275] no items to output this cycle E1121 12:12:46.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:12:46.412101 3798194 memory.go:184] no items to output this cycle I1121 12:12:46.412434 3798194 cpu.go:275] no items to output this cycle E1121 12:12:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:12:56.411957 3798194 memory.go:191] Add success. W1121 12:12:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:12:56.412003 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:12:56.412006 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:12:56.412394 3798194 cpu.go:282] Add success. I1121 12:12:56.440909 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:12:56.441443 3798194 net.go:642] Add success. I1121 12:12:56.445686 3798194 net.go:778] primary dev: ETH0 I1121 12:12:56.445703 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:12:56.445714 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:12:56.951151 3798194 disk_info.go:125] begin check local disk info of client I1121 12:12:56.955073 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:12:56.955082 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e1400 0xc0002e1440] I1121 12:12:57.445612 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:12:57.445827 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:12:57.445894 3798194 disk_worker.go:723] disk space is not compliant W1121 12:12:57.445897 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:12:57.450895 3798194 disk_worker.go:509] system disk:vda1 I1121 12:12:57.450937 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:12:58.445987 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:12:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:12:59.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:12:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:12:59.471647 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:13:06.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:13:06.411970 3798194 memory.go:184] no items to output this cycle I1121 12:13:06.412126 3798194 cpu.go:275] no items to output this cycle E1121 12:13:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:13:16.411984 3798194 memory.go:184] no items to output this cycle I1121 12:13:16.412155 3798194 cpu.go:275] no items to output this cycle E1121 12:13:26.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:13:26.413269 3798194 memory.go:191] Add success. I1121 12:13:26.417641 3798194 cpu.go:282] Add success. I1121 12:13:26.441291 3798194 net.go:642] Add success. I1121 12:13:26.442919 3798194 net.go:778] primary dev: ETH0 I1121 12:13:26.442934 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:13:26.442946 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:13:29.450470 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:13:29.450561 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:13:29.450591 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:13:36.413299 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:13:36.413322 3798194 memory.go:184] no items to output this cycle I1121 12:13:36.413469 3798194 cpu.go:275] no items to output this cycle E1121 12:13:46.412220 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:13:46.412243 3798194 memory.go:184] no items to output this cycle I1121 12:13:46.412400 3798194 cpu.go:275] no items to output this cycle E1121 12:13:56.413193 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:13:56.413228 3798194 memory.go:191] Add success. W1121 12:13:56.413260 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:13:56.413271 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:13:56.413275 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:13:56.417820 3798194 cpu.go:282] Add success. I1121 12:13:56.441207 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:13:56.441708 3798194 net.go:642] Add success. I1121 12:13:56.443381 3798194 net.go:778] primary dev: ETH0 I1121 12:13:56.443396 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:13:56.443410 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:13:56.956138 3798194 disk_info.go:125] begin check local disk info of client I1121 12:13:56.960026 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:13:56.960036 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000489b40 0xc000489b80] I1121 12:13:57.450614 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:13:57.450901 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:13:57.450961 3798194 disk_worker.go:723] disk space is not compliant W1121 12:13:57.450965 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:13:57.460176 3798194 disk_worker.go:509] system disk:vda1 I1121 12:13:57.460218 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:13:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:13:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:13:59.450465 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:13:59.450490 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:13:59.471006 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:14:06.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:14:06.413264 3798194 memory.go:184] no items to output this cycle I1121 12:14:06.413408 3798194 cpu.go:275] no items to output this cycle E1121 12:14:16.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:14:16.411995 3798194 memory.go:184] no items to output this cycle I1121 12:14:16.412161 3798194 cpu.go:275] no items to output this cycle E1121 12:14:26.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:14:26.413264 3798194 memory.go:191] Add success. I1121 12:14:26.417540 3798194 cpu.go:282] Add success. I1121 12:14:26.442089 3798194 net.go:642] Add success. I1121 12:14:26.443789 3798194 net.go:778] primary dev: ETH0 I1121 12:14:26.443803 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:14:26.443818 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:14:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:14:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:14:29.450502 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:14:36.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:14:36.412263 3798194 memory.go:184] no items to output this cycle I1121 12:14:36.412413 3798194 cpu.go:275] no items to output this cycle E1121 12:14:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:14:46.412258 3798194 memory.go:184] no items to output this cycle I1121 12:14:46.412405 3798194 cpu.go:275] no items to output this cycle E1121 12:14:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:14:56.411957 3798194 memory.go:191] Add success. W1121 12:14:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:14:56.412003 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:14:56.412006 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:14:56.412350 3798194 cpu.go:282] Add success. I1121 12:14:56.441820 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:14:56.442329 3798194 net.go:642] Add success. I1121 12:14:56.446139 3798194 net.go:778] primary dev: ETH0 I1121 12:14:56.446154 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:14:56.446168 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:14:56.962265 3798194 disk_info.go:125] begin check local disk info of client I1121 12:14:56.966196 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:14:56.966204 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00038f1c0 0xc00038f200] I1121 12:14:57.450561 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:14:57.450734 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:14:57.450792 3798194 disk_worker.go:723] disk space is not compliant W1121 12:14:57.450795 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:14:57.459165 3798194 disk_worker.go:509] system disk:vda1 I1121 12:14:57.459208 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:14:58.446012 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:14:59.450116 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:14:59.450210 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:14:59.450234 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:14:59.470726 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:15:06.412655 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:15:06.412678 3798194 memory.go:184] no items to output this cycle I1121 12:15:06.412824 3798194 cpu.go:275] no items to output this cycle E1121 12:15:16.412934 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:15:16.412958 3798194 memory.go:184] no items to output this cycle I1121 12:15:16.413102 3798194 cpu.go:275] no items to output this cycle I1121 12:15:17.460943 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:15:17.460952 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:15:26.411327 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:15:26.411366 3798194 memory.go:191] Add success. I1121 12:15:26.411711 3798194 cpu.go:282] Add success. I1121 12:15:26.441312 3798194 net.go:642] Add success. I1121 12:15:26.445380 3798194 net.go:778] primary dev: ETH0 I1121 12:15:26.445395 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:15:26.445408 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:15:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:15:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:15:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:15:36.412375 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:15:36.412398 3798194 memory.go:184] no items to output this cycle I1121 12:15:36.412539 3798194 cpu.go:275] no items to output this cycle E1121 12:15:46.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:15:46.411990 3798194 memory.go:184] no items to output this cycle I1121 12:15:46.412149 3798194 cpu.go:275] no items to output this cycle E1121 12:15:56.411235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:15:56.411268 3798194 memory.go:191] Add success. W1121 12:15:56.411295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:15:56.411307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:15:56.411310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:15:56.415973 3798194 cpu.go:282] Add success. I1121 12:15:56.444051 3798194 event_worker.go:152] Polling the log file for events... I1121 12:15:56.444292 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:15:56.444779 3798194 net.go:642] Add success. I1121 12:15:56.446415 3798194 net.go:778] primary dev: ETH0 I1121 12:15:56.446430 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:15:56.446441 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:15:56.966310 3798194 disk_info.go:125] begin check local disk info of client I1121 12:15:56.970175 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:15:56.970185 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002f1ac0 0xc0002f1b00] E1121 12:15:57.450529 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 12:15:57.450545 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:15:57.450768 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:15:57.450777 3798194 disk_worker.go:723] disk space is not compliant W1121 12:15:57.450780 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:15:57.459159 3798194 disk_worker.go:509] system disk:vda1 I1121 12:15:57.459200 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 12:15:58.448464 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 12:15:58.448478 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 12:15:59.448975 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 12:15:59.450051 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:15:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:15:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:15:59.471638 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:16:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:16:06.411982 3798194 memory.go:184] no items to output this cycle I1121 12:16:06.412134 3798194 cpu.go:275] no items to output this cycle E1121 12:16:16.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:16:16.413267 3798194 memory.go:184] no items to output this cycle I1121 12:16:16.413415 3798194 cpu.go:275] no items to output this cycle E1121 12:16:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:16:26.413264 3798194 memory.go:191] Add success. I1121 12:16:26.417693 3798194 cpu.go:282] Add success. I1121 12:16:26.442087 3798194 net.go:642] Add success. I1121 12:16:26.443679 3798194 net.go:778] primary dev: ETH0 I1121 12:16:26.443694 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:16:26.443707 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:16:29.450406 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:16:29.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:16:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:16:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:16:36.413260 3798194 memory.go:184] no items to output this cycle I1121 12:16:36.413410 3798194 cpu.go:275] no items to output this cycle E1121 12:16:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:16:46.412089 3798194 memory.go:184] no items to output this cycle I1121 12:16:46.412454 3798194 cpu.go:275] no items to output this cycle E1121 12:16:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:16:56.413268 3798194 memory.go:191] Add success. W1121 12:16:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:16:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:16:56.413313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:16:56.417781 3798194 cpu.go:282] Add success. I1121 12:16:56.441198 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:16:56.441699 3798194 net.go:642] Add success. I1121 12:16:56.443162 3798194 net.go:778] primary dev: ETH0 I1121 12:16:56.443177 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:16:56.443189 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:16:56.970290 3798194 disk_info.go:125] begin check local disk info of client I1121 12:16:56.974156 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:16:56.974164 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e1580 0xc0002e15c0] I1121 12:16:57.451007 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:16:57.451230 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:16:57.451242 3798194 disk_worker.go:723] disk space is not compliant W1121 12:16:57.451245 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:16:57.459915 3798194 disk_worker.go:509] system disk:vda1 I1121 12:16:57.459956 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:16:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:16:59.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:16:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:16:59.450502 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:16:59.471005 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:17:06.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:17:06.411949 3798194 memory.go:184] no items to output this cycle I1121 12:17:06.412103 3798194 cpu.go:275] no items to output this cycle E1121 12:17:16.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:17:16.412253 3798194 memory.go:184] no items to output this cycle I1121 12:17:16.412394 3798194 cpu.go:275] no items to output this cycle E1121 12:17:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:17:26.411954 3798194 memory.go:191] Add success. I1121 12:17:26.412315 3798194 cpu.go:282] Add success. I1121 12:17:26.441944 3798194 net.go:642] Add success. I1121 12:17:26.446137 3798194 net.go:778] primary dev: ETH0 I1121 12:17:26.446151 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:17:26.446163 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:17:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:17:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:17:29.450501 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:17:36.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:17:36.412267 3798194 memory.go:184] no items to output this cycle I1121 12:17:36.412409 3798194 cpu.go:275] no items to output this cycle E1121 12:17:46.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:17:46.412883 3798194 memory.go:184] no items to output this cycle I1121 12:17:46.413028 3798194 cpu.go:275] no items to output this cycle E1121 12:17:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:17:56.411947 3798194 memory.go:191] Add success. W1121 12:17:56.411973 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:17:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:17:56.411988 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:17:56.412319 3798194 cpu.go:282] Add success. I1121 12:17:56.441745 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:17:56.442261 3798194 net.go:642] Add success. I1121 12:17:56.446702 3798194 net.go:778] primary dev: ETH0 I1121 12:17:56.446717 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:17:56.446730 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:17:56.975136 3798194 disk_info.go:125] begin check local disk info of client I1121 12:17:56.978932 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:17:56.978941 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00042dbc0 0xc00042dc00] I1121 12:17:57.450411 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:17:57.450565 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:17:57.450616 3798194 disk_worker.go:723] disk space is not compliant W1121 12:17:57.450619 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:17:57.460088 3798194 disk_worker.go:509] system disk:vda1 I1121 12:17:57.460141 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:17:58.446014 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:17:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:17:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:17:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:17:59.471641 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:18:06.412769 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:18:06.412791 3798194 memory.go:184] no items to output this cycle I1121 12:18:06.412943 3798194 cpu.go:275] no items to output this cycle E1121 12:18:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:18:16.411979 3798194 memory.go:184] no items to output this cycle I1121 12:18:16.412137 3798194 cpu.go:275] no items to output this cycle I1121 12:18:17.461524 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:18:17.461533 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:18:26.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:18:26.412266 3798194 memory.go:191] Add success. I1121 12:18:26.416839 3798194 cpu.go:282] Add success. I1121 12:18:26.441283 3798194 net.go:642] Add success. I1121 12:18:26.442797 3798194 net.go:778] primary dev: ETH0 I1121 12:18:26.442813 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:18:26.442827 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:18:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:18:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:18:29.450123 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:18:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:18:36.411982 3798194 memory.go:184] no items to output this cycle I1121 12:18:36.412131 3798194 cpu.go:275] no items to output this cycle E1121 12:18:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:18:46.413254 3798194 memory.go:184] no items to output this cycle I1121 12:18:46.413399 3798194 cpu.go:275] no items to output this cycle E1121 12:18:56.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:18:56.412260 3798194 memory.go:191] Add success. W1121 12:18:56.412293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:18:56.412304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:18:56.412308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:18:56.416665 3798194 cpu.go:282] Add success. I1121 12:18:56.440945 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:18:56.441537 3798194 net.go:642] Add success. I1121 12:18:56.442983 3798194 net.go:778] primary dev: ETH0 I1121 12:18:56.442998 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:18:56.443011 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:18:56.979047 3798194 disk_info.go:125] begin check local disk info of client I1121 12:18:56.982938 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:18:56.982947 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b1180 0xc0003b11c0] I1121 12:18:57.451001 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:18:57.451230 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:18:57.451241 3798194 disk_worker.go:723] disk space is not compliant W1121 12:18:57.451244 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:18:57.461701 3798194 disk_worker.go:509] system disk:vda1 I1121 12:18:57.461744 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:18:58.446374 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:18:59.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:18:59.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:18:59.450492 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:18:59.470982 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:19:06.412745 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:19:06.412768 3798194 memory.go:184] no items to output this cycle I1121 12:19:06.412923 3798194 cpu.go:275] no items to output this cycle E1121 12:19:16.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:19:16.413270 3798194 memory.go:184] no items to output this cycle I1121 12:19:16.413422 3798194 cpu.go:275] no items to output this cycle E1121 12:19:26.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:19:26.413281 3798194 memory.go:191] Add success. I1121 12:19:26.417460 3798194 cpu.go:282] Add success. I1121 12:19:26.441996 3798194 net.go:642] Add success. I1121 12:19:26.443514 3798194 net.go:778] primary dev: ETH0 I1121 12:19:26.443529 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:19:26.443542 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:19:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:19:29.450489 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:19:29.450519 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:19:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:19:36.413261 3798194 memory.go:184] no items to output this cycle I1121 12:19:36.413407 3798194 cpu.go:275] no items to output this cycle E1121 12:19:46.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:19:46.413251 3798194 memory.go:184] no items to output this cycle I1121 12:19:46.413397 3798194 cpu.go:275] no items to output this cycle E1121 12:19:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:19:56.412267 3798194 memory.go:191] Add success. W1121 12:19:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:19:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:19:56.412311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:19:56.416941 3798194 cpu.go:282] Add success. I1121 12:19:56.441321 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:19:56.441871 3798194 net.go:642] Add success. I1121 12:19:56.443366 3798194 net.go:778] primary dev: ETH0 I1121 12:19:56.443381 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:19:56.443392 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:19:56.984149 3798194 disk_info.go:125] begin check local disk info of client I1121 12:19:56.988394 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:19:56.988403 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000267a80 0xc000267ac0] I1121 12:19:57.450962 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:19:57.451157 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:19:57.451212 3798194 disk_worker.go:723] disk space is not compliant W1121 12:19:57.451215 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:19:57.462031 3798194 disk_worker.go:509] system disk:vda1 I1121 12:19:57.462070 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:19:58.446197 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:19:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:19:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:19:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:19:59.471651 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:20:06.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:20:06.411973 3798194 memory.go:184] no items to output this cycle I1121 12:20:06.412123 3798194 cpu.go:275] no items to output this cycle E1121 12:20:16.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:20:16.412875 3798194 memory.go:184] no items to output this cycle I1121 12:20:16.413019 3798194 cpu.go:275] no items to output this cycle E1121 12:20:26.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:20:26.413270 3798194 memory.go:191] Add success. I1121 12:20:26.417927 3798194 cpu.go:282] Add success. I1121 12:20:26.441485 3798194 net.go:642] Add success. I1121 12:20:26.443250 3798194 net.go:778] primary dev: ETH0 I1121 12:20:26.443264 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:20:26.443276 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:20:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:20:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:20:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:20:36.412847 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:20:36.412868 3798194 memory.go:184] no items to output this cycle I1121 12:20:36.413018 3798194 cpu.go:275] no items to output this cycle E1121 12:20:46.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:20:46.411992 3798194 memory.go:184] no items to output this cycle I1121 12:20:46.412150 3798194 cpu.go:275] no items to output this cycle E1121 12:20:56.412961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:20:56.413073 3798194 memory.go:191] Add success. W1121 12:20:56.413100 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:20:56.413128 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:20:56.413133 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:20:56.417865 3798194 cpu.go:282] Add success. I1121 12:20:56.442806 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:20:56.443302 3798194 net.go:642] Add success. I1121 12:20:56.444881 3798194 net.go:778] primary dev: ETH0 I1121 12:20:56.444898 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:20:56.444912 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:20:56.990147 3798194 disk_info.go:125] begin check local disk info of client I1121 12:20:56.993907 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:20:56.993916 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071180 0xc0000711c0] E1121 12:20:57.445489 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 12:20:57.445504 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:20:57.445745 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:20:57.445755 3798194 disk_worker.go:723] disk space is not compliant W1121 12:20:57.445758 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:20:57.447853 3798194 disk_worker.go:509] system disk:vda1 I1121 12:20:57.447895 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 12:20:58.448083 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 12:20:58.448098 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 12:20:59.448625 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 12:20:59.449708 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:20:59.449762 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:20:59.449790 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:20:59.471280 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:21:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:21:06.412254 3798194 memory.go:184] no items to output this cycle I1121 12:21:06.412396 3798194 cpu.go:275] no items to output this cycle E1121 12:21:16.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:21:16.411996 3798194 memory.go:184] no items to output this cycle I1121 12:21:16.412178 3798194 cpu.go:275] no items to output this cycle I1121 12:21:17.462163 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:21:17.462171 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:21:26.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:21:26.412886 3798194 memory.go:191] Add success. I1121 12:21:26.417226 3798194 cpu.go:282] Add success. I1121 12:21:26.441744 3798194 net.go:642] Add success. I1121 12:21:26.443403 3798194 net.go:778] primary dev: ETH0 I1121 12:21:26.443419 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:21:26.443431 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:21:29.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:21:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:21:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:21:36.412819 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:21:36.412843 3798194 memory.go:184] no items to output this cycle I1121 12:21:36.412997 3798194 cpu.go:275] no items to output this cycle E1121 12:21:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:21:46.412253 3798194 memory.go:184] no items to output this cycle I1121 12:21:46.412403 3798194 cpu.go:275] no items to output this cycle E1121 12:21:56.412416 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:21:56.412451 3798194 memory.go:191] Add success. W1121 12:21:56.412483 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:21:56.412496 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:21:56.412499 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:21:56.416631 3798194 cpu.go:282] Add success. I1121 12:21:56.441157 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:21:56.441630 3798194 net.go:642] Add success. I1121 12:21:56.443294 3798194 net.go:778] primary dev: ETH0 I1121 12:21:56.443310 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:21:56.443327 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:21:56.994012 3798194 disk_info.go:125] begin check local disk info of client I1121 12:21:56.998244 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:21:56.998253 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fa900 0xc0003fa940] I1121 12:21:57.451205 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:21:57.451412 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:21:57.451423 3798194 disk_worker.go:723] disk space is not compliant W1121 12:21:57.451426 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:21:57.459911 3798194 disk_worker.go:509] system disk:vda1 I1121 12:21:57.459954 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:21:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:21:59.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:21:59.450515 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:21:59.450618 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:21:59.470727 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:22:06.413258 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:22:06.413286 3798194 memory.go:184] no items to output this cycle I1121 12:22:06.413441 3798194 cpu.go:275] no items to output this cycle E1121 12:22:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:22:16.411982 3798194 memory.go:184] no items to output this cycle I1121 12:22:16.412142 3798194 cpu.go:275] no items to output this cycle E1121 12:22:26.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:22:26.411950 3798194 memory.go:191] Add success. I1121 12:22:26.412290 3798194 cpu.go:282] Add success. I1121 12:22:26.441734 3798194 net.go:642] Add success. I1121 12:22:26.446027 3798194 net.go:778] primary dev: ETH0 I1121 12:22:26.446045 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:22:26.446057 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:22:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:22:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:22:29.450512 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:22:36.412910 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:22:36.412935 3798194 memory.go:184] no items to output this cycle I1121 12:22:36.413079 3798194 cpu.go:275] no items to output this cycle E1121 12:22:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:22:46.411987 3798194 memory.go:184] no items to output this cycle I1121 12:22:46.412140 3798194 cpu.go:275] no items to output this cycle E1121 12:22:56.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:22:56.413278 3798194 memory.go:191] Add success. W1121 12:22:56.413310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:22:56.413322 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:22:56.413325 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:22:56.417655 3798194 cpu.go:282] Add success. I1121 12:22:56.440945 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:22:56.441539 3798194 net.go:642] Add success. I1121 12:22:56.443130 3798194 net.go:778] primary dev: ETH0 I1121 12:22:56.443145 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:22:56.443158 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:22:57.000154 3798194 disk_info.go:125] begin check local disk info of client I1121 12:22:57.004316 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:22:57.004325 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000373ac0 0xc000373b00] I1121 12:22:57.451218 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:22:57.451372 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:22:57.451427 3798194 disk_worker.go:723] disk space is not compliant W1121 12:22:57.451431 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:22:57.463442 3798194 disk_worker.go:509] system disk:vda1 I1121 12:22:57.463482 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:22:58.446387 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:22:59.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:22:59.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:22:59.450500 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:22:59.471004 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:23:06.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:23:06.413261 3798194 memory.go:184] no items to output this cycle I1121 12:23:06.413407 3798194 cpu.go:275] no items to output this cycle E1121 12:23:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:23:16.411971 3798194 memory.go:184] no items to output this cycle I1121 12:23:16.412140 3798194 cpu.go:275] no items to output this cycle E1121 12:23:26.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:23:26.412261 3798194 memory.go:191] Add success. I1121 12:23:26.416577 3798194 cpu.go:282] Add success. I1121 12:23:26.442106 3798194 net.go:642] Add success. I1121 12:23:26.443731 3798194 net.go:778] primary dev: ETH0 I1121 12:23:26.443745 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:23:26.443761 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:23:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:23:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:23:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:23:36.412341 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:23:36.412431 3798194 memory.go:184] no items to output this cycle I1121 12:23:36.412761 3798194 cpu.go:275] no items to output this cycle E1121 12:23:46.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:23:46.413272 3798194 memory.go:184] no items to output this cycle I1121 12:23:46.413424 3798194 cpu.go:275] no items to output this cycle E1121 12:23:56.412427 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:23:56.412463 3798194 memory.go:191] Add success. W1121 12:23:56.412488 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:23:56.412499 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:23:56.412503 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:23:56.417153 3798194 cpu.go:282] Add success. I1121 12:23:56.441554 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:23:56.442050 3798194 net.go:642] Add success. I1121 12:23:56.443579 3798194 net.go:778] primary dev: ETH0 I1121 12:23:56.443593 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:23:56.443617 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:23:57.006146 3798194 disk_info.go:125] begin check local disk info of client I1121 12:23:57.010021 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:23:57.010030 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a51c0 0xc0003a5200] I1121 12:23:57.450925 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:23:57.451149 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:23:57.451160 3798194 disk_worker.go:723] disk space is not compliant W1121 12:23:57.451163 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:23:57.459930 3798194 disk_worker.go:509] system disk:vda1 I1121 12:23:57.459971 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:23:58.446551 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:23:59.450616 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:23:59.450686 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:23:59.450715 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:23:59.471195 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:24:06.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:24:06.411981 3798194 memory.go:184] no items to output this cycle I1121 12:24:06.412143 3798194 cpu.go:275] no items to output this cycle E1121 12:24:16.412566 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:24:16.412589 3798194 memory.go:184] no items to output this cycle I1121 12:24:16.412745 3798194 cpu.go:275] no items to output this cycle I1121 12:24:17.462661 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:24:17.462670 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:24:26.411396 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:24:26.411430 3798194 memory.go:191] Add success. I1121 12:24:26.411732 3798194 cpu.go:282] Add success. I1121 12:24:26.441309 3798194 net.go:642] Add success. I1121 12:24:26.445754 3798194 net.go:778] primary dev: ETH0 I1121 12:24:26.445768 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:24:26.445782 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:24:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:24:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:24:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:24:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:24:36.413257 3798194 memory.go:184] no items to output this cycle I1121 12:24:36.413404 3798194 cpu.go:275] no items to output this cycle E1121 12:24:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:24:46.413254 3798194 memory.go:184] no items to output this cycle I1121 12:24:46.413400 3798194 cpu.go:275] no items to output this cycle E1121 12:24:56.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:24:56.413266 3798194 memory.go:191] Add success. W1121 12:24:56.413293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:24:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:24:56.413308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:24:56.417589 3798194 cpu.go:282] Add success. I1121 12:24:56.440910 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:24:56.441509 3798194 net.go:642] Add success. I1121 12:24:56.443219 3798194 net.go:778] primary dev: ETH0 I1121 12:24:56.443233 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:24:56.443245 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:24:57.012138 3798194 disk_info.go:125] begin check local disk info of client I1121 12:24:57.016472 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:24:57.016481 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000482e80 0xc000482ec0] I1121 12:24:57.445829 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:24:57.446098 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:24:57.446123 3798194 disk_worker.go:723] disk space is not compliant W1121 12:24:57.446126 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:24:57.448989 3798194 disk_worker.go:509] system disk:vda1 I1121 12:24:57.449026 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:24:58.446386 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:24:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:24:59.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:24:59.450496 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:24:59.470993 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:25:06.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:25:06.413259 3798194 memory.go:184] no items to output this cycle I1121 12:25:06.413409 3798194 cpu.go:275] no items to output this cycle E1121 12:25:16.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:25:16.413274 3798194 memory.go:184] no items to output this cycle I1121 12:25:16.413428 3798194 cpu.go:275] no items to output this cycle E1121 12:25:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:25:26.412279 3798194 memory.go:191] Add success. I1121 12:25:26.416638 3798194 cpu.go:282] Add success. I1121 12:25:26.442195 3798194 net.go:642] Add success. I1121 12:25:26.443672 3798194 net.go:778] primary dev: ETH0 I1121 12:25:26.443687 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:25:26.443701 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:25:29.450035 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:25:29.450128 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:25:29.450154 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:25:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:25:36.413261 3798194 memory.go:184] no items to output this cycle I1121 12:25:36.413403 3798194 cpu.go:275] no items to output this cycle E1121 12:25:46.413410 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:25:46.413431 3798194 memory.go:184] no items to output this cycle I1121 12:25:46.413578 3798194 cpu.go:275] no items to output this cycle E1121 12:25:56.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:25:56.412262 3798194 memory.go:191] Add success. W1121 12:25:56.412288 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:25:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:25:56.412303 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:25:56.416585 3798194 cpu.go:282] Add success. I1121 12:25:56.441638 3798194 event_worker.go:152] Polling the log file for events... I1121 12:25:56.444785 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:25:56.445279 3798194 net.go:642] Add success. I1121 12:25:56.446785 3798194 net.go:778] primary dev: ETH0 I1121 12:25:56.446799 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:25:56.446811 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:25:57.019144 3798194 disk_info.go:125] begin check local disk info of client I1121 12:25:57.022951 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:25:57.022961 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a3200 0xc0003a3240] E1121 12:25:57.450875 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 12:25:57.450889 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:25:57.451086 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:25:57.451097 3798194 disk_worker.go:723] disk space is not compliant W1121 12:25:57.451100 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:25:57.459883 3798194 disk_worker.go:509] system disk:vda1 I1121 12:25:57.459925 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 12:25:58.448483 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 12:25:58.448498 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 12:25:59.448972 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 12:25:59.450046 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:25:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:25:59.450123 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:25:59.471645 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:26:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:26:06.411981 3798194 memory.go:184] no items to output this cycle I1121 12:26:06.412131 3798194 cpu.go:275] no items to output this cycle E1121 12:26:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:26:16.412259 3798194 memory.go:184] no items to output this cycle I1121 12:26:16.412423 3798194 cpu.go:275] no items to output this cycle E1121 12:26:26.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:26:26.413264 3798194 memory.go:191] Add success. I1121 12:26:26.417532 3798194 cpu.go:282] Add success. I1121 12:26:26.441749 3798194 net.go:642] Add success. I1121 12:26:26.443262 3798194 net.go:778] primary dev: ETH0 I1121 12:26:26.443278 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:26:26.443292 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:26:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:26:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:26:29.450144 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:26:36.411944 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:26:36.411965 3798194 memory.go:184] no items to output this cycle I1121 12:26:36.412132 3798194 cpu.go:275] no items to output this cycle E1121 12:26:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:26:46.411975 3798194 memory.go:184] no items to output this cycle I1121 12:26:46.412131 3798194 cpu.go:275] no items to output this cycle E1121 12:26:56.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:26:56.412263 3798194 memory.go:191] Add success. W1121 12:26:56.412296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:26:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:26:56.412310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:26:56.416810 3798194 cpu.go:282] Add success. I1121 12:26:56.441184 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:26:56.441654 3798194 net.go:642] Add success. I1121 12:26:56.443378 3798194 net.go:778] primary dev: ETH0 I1121 12:26:56.443392 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:26:56.443404 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:26:57.023069 3798194 disk_info.go:125] begin check local disk info of client I1121 12:26:57.027316 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:26:57.027325 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000457740 0xc000457780] I1121 12:26:57.451181 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:26:57.451391 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:26:57.451402 3798194 disk_worker.go:723] disk space is not compliant W1121 12:26:57.451405 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:26:57.460848 3798194 disk_worker.go:509] system disk:vda1 I1121 12:26:57.460894 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:26:58.446199 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:26:59.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:26:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:26:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:26:59.471642 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:27:06.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:27:06.413259 3798194 memory.go:184] no items to output this cycle I1121 12:27:06.413412 3798194 cpu.go:275] no items to output this cycle E1121 12:27:16.412478 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:27:16.412507 3798194 memory.go:184] no items to output this cycle I1121 12:27:16.412731 3798194 cpu.go:275] no items to output this cycle I1121 12:27:17.462781 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:27:17.462790 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:27:26.412806 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:27:26.412841 3798194 memory.go:191] Add success. I1121 12:27:26.417271 3798194 cpu.go:282] Add success. I1121 12:27:26.441741 3798194 net.go:642] Add success. I1121 12:27:26.443468 3798194 net.go:778] primary dev: ETH0 I1121 12:27:26.443482 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:27:26.443496 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:27:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:27:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:27:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:27:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:27:36.413257 3798194 memory.go:184] no items to output this cycle I1121 12:27:36.413403 3798194 cpu.go:275] no items to output this cycle E1121 12:27:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:27:46.412257 3798194 memory.go:184] no items to output this cycle I1121 12:27:46.412408 3798194 cpu.go:275] no items to output this cycle E1121 12:27:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:27:56.412266 3798194 memory.go:191] Add success. W1121 12:27:56.412294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:27:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:27:56.412309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:27:56.416430 3798194 cpu.go:282] Add success. I1121 12:27:56.440953 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:27:56.441543 3798194 net.go:642] Add success. I1121 12:27:56.443147 3798194 net.go:778] primary dev: ETH0 I1121 12:27:56.443161 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:27:56.443174 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:27:57.029147 3798194 disk_info.go:125] begin check local disk info of client I1121 12:27:57.032705 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:27:57.032714 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004c3a00 0xc0004c3a40] I1121 12:27:57.450631 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:27:57.450833 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:27:57.450845 3798194 disk_worker.go:723] disk space is not compliant W1121 12:27:57.450848 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:27:57.458991 3798194 disk_worker.go:509] system disk:vda1 I1121 12:27:57.459030 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:27:58.446389 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:27:59.450497 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:27:59.450572 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:27:59.450601 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:27:59.471118 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:28:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:28:06.412255 3798194 memory.go:184] no items to output this cycle I1121 12:28:06.412395 3798194 cpu.go:275] no items to output this cycle E1121 12:28:16.412251 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:28:16.412276 3798194 memory.go:184] no items to output this cycle I1121 12:28:16.412431 3798194 cpu.go:275] no items to output this cycle E1121 12:28:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:28:26.411948 3798194 memory.go:191] Add success. I1121 12:28:26.412277 3798194 cpu.go:282] Add success. I1121 12:28:26.441754 3798194 net.go:642] Add success. I1121 12:28:26.445594 3798194 net.go:778] primary dev: ETH0 I1121 12:28:26.445610 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:28:26.445621 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:28:29.450002 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:28:29.450082 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:28:29.450120 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:28:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:28:36.412256 3798194 memory.go:184] no items to output this cycle I1121 12:28:36.412399 3798194 cpu.go:275] no items to output this cycle E1121 12:28:46.413250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:28:46.413273 3798194 memory.go:184] no items to output this cycle I1121 12:28:46.413431 3798194 cpu.go:275] no items to output this cycle E1121 12:28:56.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:28:56.412272 3798194 memory.go:191] Add success. W1121 12:28:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:28:56.412316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:28:56.412319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:28:56.416300 3798194 cpu.go:282] Add success. I1121 12:28:56.441708 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:28:56.442314 3798194 net.go:642] Add success. I1121 12:28:56.443778 3798194 net.go:778] primary dev: ETH0 I1121 12:28:56.443792 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:28:56.443805 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:28:57.035433 3798194 disk_info.go:125] begin check local disk info of client I1121 12:28:57.038946 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:28:57.038954 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040d400 0xc00040d440] I1121 12:28:57.450631 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:28:57.450854 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:28:57.450865 3798194 disk_worker.go:723] disk space is not compliant W1121 12:28:57.450868 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:28:57.459063 3798194 disk_worker.go:509] system disk:vda1 I1121 12:28:57.459106 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:28:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:28:59.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:28:59.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:28:59.450498 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:28:59.471025 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:29:06.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:29:06.413270 3798194 memory.go:184] no items to output this cycle I1121 12:29:06.413415 3798194 cpu.go:275] no items to output this cycle E1121 12:29:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:29:16.411987 3798194 memory.go:184] no items to output this cycle I1121 12:29:16.412146 3798194 cpu.go:275] no items to output this cycle E1121 12:29:26.412255 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:29:26.412300 3798194 memory.go:191] Add success. I1121 12:29:26.416817 3798194 cpu.go:282] Add success. I1121 12:29:26.441289 3798194 net.go:642] Add success. I1121 12:29:26.442810 3798194 net.go:778] primary dev: ETH0 I1121 12:29:26.442825 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:29:26.442838 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:29:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:29:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:29:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:29:36.413012 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:29:36.413036 3798194 memory.go:184] no items to output this cycle I1121 12:29:36.413197 3798194 cpu.go:275] no items to output this cycle E1121 12:29:46.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:29:46.412262 3798194 memory.go:184] no items to output this cycle I1121 12:29:46.412406 3798194 cpu.go:275] no items to output this cycle E1121 12:29:56.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:29:56.412279 3798194 memory.go:191] Add success. W1121 12:29:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:29:56.412321 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:29:56.412325 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:29:56.416698 3798194 cpu.go:282] Add success. I1121 12:29:56.441076 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:29:56.441659 3798194 net.go:642] Add success. I1121 12:29:56.443180 3798194 net.go:778] primary dev: ETH0 I1121 12:29:56.443203 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:29:56.443217 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:29:57.042157 3798194 disk_info.go:125] begin check local disk info of client I1121 12:29:57.063496 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:29:57.063507 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000354a00 0xc000354a40] I1121 12:29:57.450272 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:29:57.450444 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:29:57.450498 3798194 disk_worker.go:723] disk space is not compliant W1121 12:29:57.450501 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:29:57.458593 3798194 disk_worker.go:509] system disk:vda1 I1121 12:29:57.458634 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:29:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:29:59.450122 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:29:59.450200 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:29:59.450230 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:29:59.470694 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:30:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:30:06.411976 3798194 memory.go:184] no items to output this cycle I1121 12:30:06.412127 3798194 cpu.go:275] no items to output this cycle E1121 12:30:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:30:16.412260 3798194 memory.go:184] no items to output this cycle I1121 12:30:16.412407 3798194 cpu.go:275] no items to output this cycle I1121 12:30:17.463707 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:30:17.463715 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:30:26.411471 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:30:26.411503 3798194 memory.go:191] Add success. I1121 12:30:26.411823 3798194 cpu.go:282] Add success. I1121 12:30:26.441442 3798194 net.go:642] Add success. I1121 12:30:26.445914 3798194 net.go:778] primary dev: ETH0 I1121 12:30:26.445929 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:30:26.445942 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:30:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:30:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:30:29.450498 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:30:36.412458 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:30:36.412480 3798194 memory.go:184] no items to output this cycle I1121 12:30:36.412621 3798194 cpu.go:275] no items to output this cycle E1121 12:30:46.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:30:46.412882 3798194 memory.go:184] no items to output this cycle I1121 12:30:46.413247 3798194 cpu.go:275] no items to output this cycle E1121 12:30:56.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:30:56.412260 3798194 memory.go:191] Add success. W1121 12:30:56.412291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:30:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:30:56.412306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:30:56.417516 3798194 cpu.go:282] Add success. I1121 12:30:56.444487 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:30:56.445076 3798194 net.go:642] Add success. I1121 12:30:56.446689 3798194 net.go:778] primary dev: ETH0 I1121 12:30:56.446704 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:30:56.446718 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:30:57.063615 3798194 disk_info.go:125] begin check local disk info of client I1121 12:30:57.067198 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:30:57.067206 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000354440 0xc000354480] E1121 12:30:57.450293 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 12:30:57.450308 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:30:57.450524 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:30:57.450534 3798194 disk_worker.go:723] disk space is not compliant W1121 12:30:57.450537 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:30:57.460254 3798194 disk_worker.go:509] system disk:vda1 I1121 12:30:57.460298 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 12:30:58.447793 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 12:30:58.447809 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 12:30:59.448507 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 12:30:59.450638 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:30:59.450690 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:30:59.450709 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:30:59.471220 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:31:06.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:31:06.411979 3798194 memory.go:184] no items to output this cycle I1121 12:31:06.412137 3798194 cpu.go:275] no items to output this cycle E1121 12:31:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:31:16.413266 3798194 memory.go:184] no items to output this cycle I1121 12:31:16.413435 3798194 cpu.go:275] no items to output this cycle E1121 12:31:26.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:31:26.411958 3798194 memory.go:191] Add success. I1121 12:31:26.412293 3798194 cpu.go:282] Add success. I1121 12:31:26.441970 3798194 net.go:642] Add success. I1121 12:31:26.445970 3798194 net.go:778] primary dev: ETH0 I1121 12:31:26.445985 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:31:26.445998 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:31:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:31:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:31:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:31:36.412915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:31:36.412938 3798194 memory.go:184] no items to output this cycle I1121 12:31:36.413079 3798194 cpu.go:275] no items to output this cycle E1121 12:31:46.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:31:46.413260 3798194 memory.go:184] no items to output this cycle I1121 12:31:46.413410 3798194 cpu.go:275] no items to output this cycle E1121 12:31:56.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:31:56.412894 3798194 memory.go:191] Add success. W1121 12:31:56.412920 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:31:56.412932 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:31:56.412935 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:31:56.417374 3798194 cpu.go:282] Add success. I1121 12:31:56.440948 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:31:56.441525 3798194 net.go:642] Add success. I1121 12:31:56.443319 3798194 net.go:778] primary dev: ETH0 I1121 12:31:56.443335 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:31:56.443349 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:31:57.069139 3798194 disk_info.go:125] begin check local disk info of client I1121 12:31:57.073213 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:31:57.073221 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000451980 0xc0004519c0] I1121 12:31:57.450559 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:31:57.450766 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:31:57.450777 3798194 disk_worker.go:723] disk space is not compliant W1121 12:31:57.450780 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:31:57.459265 3798194 disk_worker.go:509] system disk:vda1 I1121 12:31:57.459306 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:31:58.446102 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:31:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:31:59.450127 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:31:59.450157 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:31:59.471671 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:32:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:32:06.412255 3798194 memory.go:184] no items to output this cycle I1121 12:32:06.412401 3798194 cpu.go:275] no items to output this cycle E1121 12:32:16.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:32:16.413257 3798194 memory.go:184] no items to output this cycle I1121 12:32:16.413403 3798194 cpu.go:275] no items to output this cycle E1121 12:32:26.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:32:26.413270 3798194 memory.go:191] Add success. I1121 12:32:26.417605 3798194 cpu.go:282] Add success. I1121 12:32:26.442260 3798194 net.go:642] Add success. I1121 12:32:26.443745 3798194 net.go:778] primary dev: ETH0 I1121 12:32:26.443761 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:32:26.443779 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:32:29.449657 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:32:29.449731 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:32:29.449757 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:32:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:32:36.412262 3798194 memory.go:184] no items to output this cycle I1121 12:32:36.412403 3798194 cpu.go:275] no items to output this cycle E1121 12:32:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:32:46.413257 3798194 memory.go:184] no items to output this cycle I1121 12:32:46.413417 3798194 cpu.go:275] no items to output this cycle E1121 12:32:56.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:32:56.413274 3798194 memory.go:191] Add success. W1121 12:32:56.413300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:32:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:32:56.413316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:32:56.417623 3798194 cpu.go:282] Add success. I1121 12:32:56.441100 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:32:56.441581 3798194 net.go:642] Add success. I1121 12:32:56.443340 3798194 net.go:778] primary dev: ETH0 I1121 12:32:56.443356 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:32:56.443369 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:32:57.073331 3798194 disk_info.go:125] begin check local disk info of client I1121 12:32:57.076826 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:32:57.076835 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00038a800 0xc00038a840] I1121 12:32:57.450676 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:32:57.450842 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:32:57.450895 3798194 disk_worker.go:723] disk space is not compliant W1121 12:32:57.450898 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:32:57.458963 3798194 disk_worker.go:509] system disk:vda1 I1121 12:32:57.459006 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:32:58.446400 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:32:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:32:59.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:32:59.450149 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:32:59.471657 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:33:06.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:33:06.413257 3798194 memory.go:184] no items to output this cycle I1121 12:33:06.413399 3798194 cpu.go:275] no items to output this cycle E1121 12:33:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:33:16.412269 3798194 memory.go:184] no items to output this cycle I1121 12:33:16.412422 3798194 cpu.go:275] no items to output this cycle I1121 12:33:17.464538 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:33:17.464546 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:33:26.411380 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:33:26.411412 3798194 memory.go:191] Add success. I1121 12:33:26.411747 3798194 cpu.go:282] Add success. I1121 12:33:26.441510 3798194 net.go:642] Add success. I1121 12:33:26.445760 3798194 net.go:778] primary dev: ETH0 I1121 12:33:26.445775 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:33:26.445894 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:33:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:33:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:33:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:33:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:33:36.413255 3798194 memory.go:184] no items to output this cycle I1121 12:33:36.413401 3798194 cpu.go:275] no items to output this cycle E1121 12:33:46.412342 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:33:46.412365 3798194 memory.go:184] no items to output this cycle I1121 12:33:46.412517 3798194 cpu.go:275] no items to output this cycle E1121 12:33:56.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:33:56.413270 3798194 memory.go:191] Add success. W1121 12:33:56.413301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:33:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:33:56.413316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:33:56.417704 3798194 cpu.go:282] Add success. I1121 12:33:56.441209 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:33:56.441674 3798194 net.go:642] Add success. I1121 12:33:56.443163 3798194 net.go:778] primary dev: ETH0 I1121 12:33:56.443177 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:33:56.443190 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:33:57.079977 3798194 disk_info.go:125] begin check local disk info of client I1121 12:33:57.083591 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:33:57.083600 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a3d00 0xc0003a3d40] I1121 12:33:57.450624 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:33:57.450785 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:33:57.450852 3798194 disk_worker.go:723] disk space is not compliant W1121 12:33:57.450856 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:33:57.458983 3798194 disk_worker.go:509] system disk:vda1 I1121 12:33:57.459024 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:33:58.446385 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:33:59.450498 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:33:59.450574 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:33:59.450605 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:33:59.471134 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:34:06.411992 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:34:06.412025 3798194 memory.go:184] no items to output this cycle I1121 12:34:06.412195 3798194 cpu.go:275] no items to output this cycle E1121 12:34:16.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:34:16.412258 3798194 memory.go:184] no items to output this cycle I1121 12:34:16.412411 3798194 cpu.go:275] no items to output this cycle E1121 12:34:26.413002 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:34:26.413036 3798194 memory.go:191] Add success. I1121 12:34:26.417495 3798194 cpu.go:282] Add success. I1121 12:34:26.442191 3798194 net.go:642] Add success. I1121 12:34:26.443813 3798194 net.go:778] primary dev: ETH0 I1121 12:34:26.443828 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:34:26.443841 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:34:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:34:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:34:29.450502 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:34:36.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:34:36.411977 3798194 memory.go:184] no items to output this cycle I1121 12:34:36.412126 3798194 cpu.go:275] no items to output this cycle E1121 12:34:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:34:46.413254 3798194 memory.go:184] no items to output this cycle I1121 12:34:46.413401 3798194 cpu.go:275] no items to output this cycle E1121 12:34:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:34:56.412268 3798194 memory.go:191] Add success. W1121 12:34:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:34:56.412321 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:34:56.412325 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:34:56.416746 3798194 cpu.go:282] Add success. I1121 12:34:56.441326 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:34:56.441801 3798194 net.go:642] Add success. I1121 12:34:56.444758 3798194 net.go:778] primary dev: ETH0 I1121 12:34:56.444773 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:34:56.444785 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:34:57.083711 3798194 disk_info.go:125] begin check local disk info of client I1121 12:34:57.087303 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:34:57.087312 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004c01c0 0xc0004c0200] I1121 12:34:57.450669 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:34:57.450832 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:34:57.450892 3798194 disk_worker.go:723] disk space is not compliant W1121 12:34:57.450895 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:34:57.459146 3798194 disk_worker.go:509] system disk:vda1 I1121 12:34:57.459185 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:34:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:34:59.450044 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:34:59.450144 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:34:59.450169 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:34:59.471682 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:35:06.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:35:06.413256 3798194 memory.go:184] no items to output this cycle I1121 12:35:06.413396 3798194 cpu.go:275] no items to output this cycle E1121 12:35:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:35:16.411978 3798194 memory.go:184] no items to output this cycle I1121 12:35:16.412138 3798194 cpu.go:275] no items to output this cycle E1121 12:35:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:35:26.413267 3798194 memory.go:191] Add success. I1121 12:35:26.417313 3798194 cpu.go:282] Add success. I1121 12:35:26.441761 3798194 net.go:642] Add success. I1121 12:35:26.443203 3798194 net.go:778] primary dev: ETH0 I1121 12:35:26.443217 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:35:26.443233 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:35:29.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:35:29.450118 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:35:29.450145 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:35:36.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:35:36.411989 3798194 memory.go:184] no items to output this cycle I1121 12:35:36.412144 3798194 cpu.go:275] no items to output this cycle E1121 12:35:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:35:46.411976 3798194 memory.go:184] no items to output this cycle I1121 12:35:46.412143 3798194 cpu.go:275] no items to output this cycle E1121 12:35:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:35:56.412266 3798194 memory.go:191] Add success. W1121 12:35:56.412294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:35:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:35:56.412308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:35:56.416720 3798194 cpu.go:282] Add success. I1121 12:35:56.444588 3798194 event_worker.go:152] Polling the log file for events... I1121 12:35:56.444943 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:35:56.445487 3798194 net.go:642] Add success. I1121 12:35:56.447026 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795233956, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 12:35:56.447146 3798194 net.go:778] primary dev: ETH0 I1121 12:35:56.447159 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:35:56.447172 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:35:56.456335 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 I1121 12:35:57.095149 3798194 disk_info.go:125] begin check local disk info of client I1121 12:35:57.105639 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:35:57.105648 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d7d80 0xc0003d7dc0] E1121 12:35:57.445636 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 12:35:57.445652 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:35:57.445922 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:35:57.445933 3798194 disk_worker.go:723] disk space is not compliant W1121 12:35:57.445937 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:35:57.448411 3798194 disk_worker.go:509] system disk:vda1 I1121 12:35:57.448453 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 12:35:58.448187 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 12:35:58.448201 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:35:59.449993 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:35:59.450045 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:35:59.450064 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:35:59.450184 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 12:35:59.471579 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:36:06.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:36:06.413260 3798194 memory.go:184] no items to output this cycle I1121 12:36:06.413410 3798194 cpu.go:275] no items to output this cycle E1121 12:36:16.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:36:16.411992 3798194 memory.go:184] no items to output this cycle I1121 12:36:16.412150 3798194 cpu.go:275] no items to output this cycle I1121 12:36:17.465569 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:36:17.465578 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:36:26.413254 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:36:26.413292 3798194 memory.go:191] Add success. I1121 12:36:26.417676 3798194 cpu.go:282] Add success. I1121 12:36:26.441275 3798194 net.go:642] Add success. I1121 12:36:26.442844 3798194 net.go:778] primary dev: ETH0 I1121 12:36:26.442860 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:36:26.442872 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:36:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:36:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:36:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:36:36.412878 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:36:36.412902 3798194 memory.go:184] no items to output this cycle I1121 12:36:36.413062 3798194 cpu.go:275] no items to output this cycle E1121 12:36:46.411972 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:36:46.411995 3798194 memory.go:184] no items to output this cycle I1121 12:36:46.412179 3798194 cpu.go:275] no items to output this cycle E1121 12:36:56.411930 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:36:56.411963 3798194 memory.go:191] Add success. W1121 12:36:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:36:56.412006 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:36:56.412010 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:36:56.412395 3798194 cpu.go:282] Add success. I1121 12:36:56.441870 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:36:56.442402 3798194 net.go:642] Add success. I1121 12:36:56.446647 3798194 net.go:778] primary dev: ETH0 I1121 12:36:56.446661 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:36:56.446673 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:36:57.105753 3798194 disk_info.go:125] begin check local disk info of client I1121 12:36:57.110427 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:36:57.110435 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000131b00 0xc000131b40] I1121 12:36:57.450585 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:36:57.450793 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:36:57.450804 3798194 disk_worker.go:723] disk space is not compliant W1121 12:36:57.450807 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:36:57.459452 3798194 disk_worker.go:509] system disk:vda1 I1121 12:36:57.459493 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:36:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:36:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:36:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:36:59.450140 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:36:59.471646 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:37:06.412995 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:37:06.413018 3798194 memory.go:184] no items to output this cycle I1121 12:37:06.413178 3798194 cpu.go:275] no items to output this cycle E1121 12:37:16.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:37:16.411986 3798194 memory.go:184] no items to output this cycle I1121 12:37:16.412152 3798194 cpu.go:275] no items to output this cycle E1121 12:37:26.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:37:26.411949 3798194 memory.go:191] Add success. I1121 12:37:26.412300 3798194 cpu.go:282] Add success. I1121 12:37:26.441918 3798194 net.go:642] Add success. I1121 12:37:26.445944 3798194 net.go:778] primary dev: ETH0 I1121 12:37:26.445959 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:37:26.445975 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:37:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:37:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:37:29.450502 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:37:36.412760 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:37:36.412782 3798194 memory.go:184] no items to output this cycle I1121 12:37:36.412946 3798194 cpu.go:275] no items to output this cycle E1121 12:37:46.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:37:46.412874 3798194 memory.go:184] no items to output this cycle I1121 12:37:46.413017 3798194 cpu.go:275] no items to output this cycle E1121 12:37:56.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:37:56.412281 3798194 memory.go:191] Add success. W1121 12:37:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:37:56.412323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:37:56.412327 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:37:56.417339 3798194 cpu.go:282] Add success. I1121 12:37:56.441747 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:37:56.442258 3798194 net.go:642] Add success. I1121 12:37:56.443816 3798194 net.go:778] primary dev: ETH0 I1121 12:37:56.443831 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:37:56.443844 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:37:57.112144 3798194 disk_info.go:125] begin check local disk info of client I1121 12:37:57.118797 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:37:57.118807 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030f440 0xc00030f480] I1121 12:37:57.450571 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:37:57.450771 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:37:57.450782 3798194 disk_worker.go:723] disk space is not compliant W1121 12:37:57.450785 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:37:57.461944 3798194 disk_worker.go:509] system disk:vda1 I1121 12:37:57.461987 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:37:58.446009 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:37:59.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:37:59.450488 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:37:59.450513 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:37:59.471025 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:38:06.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:38:06.411969 3798194 memory.go:184] no items to output this cycle I1121 12:38:06.412133 3798194 cpu.go:275] no items to output this cycle E1121 12:38:16.412269 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:38:16.412292 3798194 memory.go:184] no items to output this cycle I1121 12:38:16.412448 3798194 cpu.go:275] no items to output this cycle E1121 12:38:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:38:26.413266 3798194 memory.go:191] Add success. I1121 12:38:26.417392 3798194 cpu.go:282] Add success. I1121 12:38:26.441922 3798194 net.go:642] Add success. I1121 12:38:26.443510 3798194 net.go:778] primary dev: ETH0 I1121 12:38:26.443524 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:38:26.443537 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:38:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:38:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:38:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:38:36.413296 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:38:36.413325 3798194 memory.go:184] no items to output this cycle I1121 12:38:36.413524 3798194 cpu.go:275] no items to output this cycle E1121 12:38:46.413046 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:38:46.413067 3798194 memory.go:184] no items to output this cycle I1121 12:38:46.413237 3798194 cpu.go:275] no items to output this cycle E1121 12:38:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:38:56.411945 3798194 memory.go:191] Add success. W1121 12:38:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:38:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:38:56.411991 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:38:56.412329 3798194 cpu.go:282] Add success. I1121 12:38:56.441781 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:38:56.442316 3798194 net.go:642] Add success. I1121 12:38:56.446532 3798194 net.go:778] primary dev: ETH0 I1121 12:38:56.446548 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:38:56.446563 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:38:57.118900 3798194 disk_info.go:125] begin check local disk info of client I1121 12:38:57.122532 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:38:57.122542 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00042cf40 0xc00042cf80] I1121 12:38:57.451243 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:38:57.451412 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:38:57.451465 3798194 disk_worker.go:723] disk space is not compliant W1121 12:38:57.451468 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:38:57.462627 3798194 disk_worker.go:509] system disk:vda1 I1121 12:38:57.462669 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:38:58.446204 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:38:59.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:38:59.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:38:59.450510 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:38:59.471014 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:39:06.412786 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:39:06.412815 3798194 memory.go:184] no items to output this cycle I1121 12:39:06.412963 3798194 cpu.go:275] no items to output this cycle E1121 12:39:16.411984 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:39:16.412007 3798194 memory.go:184] no items to output this cycle I1121 12:39:16.412170 3798194 cpu.go:275] no items to output this cycle I1121 12:39:17.466191 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:39:17.466199 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:39:26.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:39:26.412270 3798194 memory.go:191] Add success. I1121 12:39:26.416539 3798194 cpu.go:282] Add success. I1121 12:39:26.442233 3798194 net.go:642] Add success. I1121 12:39:26.443861 3798194 net.go:778] primary dev: ETH0 I1121 12:39:26.443879 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:39:26.443892 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:39:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:39:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:39:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:39:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:39:36.413256 3798194 memory.go:184] no items to output this cycle I1121 12:39:36.413396 3798194 cpu.go:275] no items to output this cycle E1121 12:39:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:39:46.411978 3798194 memory.go:184] no items to output this cycle I1121 12:39:46.412144 3798194 cpu.go:275] no items to output this cycle E1121 12:39:56.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:39:56.413278 3798194 memory.go:191] Add success. W1121 12:39:56.413309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:39:56.413321 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:39:56.413324 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:39:56.418073 3798194 cpu.go:282] Add success. I1121 12:39:56.441574 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:39:56.442049 3798194 net.go:642] Add success. I1121 12:39:56.443652 3798194 net.go:778] primary dev: ETH0 I1121 12:39:56.443668 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:39:56.443679 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:39:57.122649 3798194 disk_info.go:125] begin check local disk info of client I1121 12:39:57.126311 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:39:57.126320 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036ddc0 0xc00036de00] I1121 12:39:57.445686 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:39:57.445961 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:39:57.445973 3798194 disk_worker.go:723] disk space is not compliant W1121 12:39:57.445976 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:39:57.448526 3798194 disk_worker.go:509] system disk:vda1 I1121 12:39:57.448592 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:39:58.446383 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:39:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:39:59.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:39:59.450497 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:39:59.471004 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:40:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:40:06.412255 3798194 memory.go:184] no items to output this cycle I1121 12:40:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 12:40:16.412652 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:40:16.412679 3798194 memory.go:184] no items to output this cycle I1121 12:40:16.412826 3798194 cpu.go:275] no items to output this cycle E1121 12:40:26.412524 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:40:26.412557 3798194 memory.go:191] Add success. I1121 12:40:26.417159 3798194 cpu.go:282] Add success. I1121 12:40:26.441700 3798194 net.go:642] Add success. I1121 12:40:26.443421 3798194 net.go:778] primary dev: ETH0 I1121 12:40:26.443435 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:40:26.443448 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:40:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:40:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:40:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:40:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:40:36.412258 3798194 memory.go:184] no items to output this cycle I1121 12:40:36.412400 3798194 cpu.go:275] no items to output this cycle E1121 12:40:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:40:46.411979 3798194 memory.go:184] no items to output this cycle I1121 12:40:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 12:40:56.412724 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:40:56.412759 3798194 memory.go:191] Add success. W1121 12:40:56.412790 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:40:56.412802 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:40:56.412805 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:40:56.417236 3798194 cpu.go:282] Add success. I1121 12:40:56.444154 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:40:56.444618 3798194 net.go:642] Add success. I1121 12:40:56.446203 3798194 net.go:778] primary dev: ETH0 I1121 12:40:56.446217 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:40:56.446230 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:40:57.126426 3798194 disk_info.go:125] begin check local disk info of client I1121 12:40:57.130030 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:40:57.130039 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000347040 0xc000347080] E1121 12:40:57.450641 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 12:40:57.450654 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:40:57.450870 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:40:57.450881 3798194 disk_worker.go:723] disk space is not compliant W1121 12:40:57.450884 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:40:57.458997 3798194 disk_worker.go:509] system disk:vda1 I1121 12:40:57.459041 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 12:40:58.447801 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 12:40:58.447816 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 12:40:59.448601 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 12:40:59.449675 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:40:59.449724 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:40:59.449751 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:40:59.471189 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:41:06.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:41:06.411975 3798194 memory.go:184] no items to output this cycle I1121 12:41:06.412131 3798194 cpu.go:275] no items to output this cycle E1121 12:41:16.413256 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:41:16.413277 3798194 memory.go:184] no items to output this cycle I1121 12:41:16.413420 3798194 cpu.go:275] no items to output this cycle E1121 12:41:26.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:41:26.412276 3798194 memory.go:191] Add success. I1121 12:41:26.416792 3798194 cpu.go:282] Add success. I1121 12:41:26.441508 3798194 net.go:642] Add success. I1121 12:41:26.442891 3798194 net.go:778] primary dev: ETH0 I1121 12:41:26.442906 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:41:26.442918 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:41:29.450198 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:41:29.450273 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:41:29.450299 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:41:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:41:36.413261 3798194 memory.go:184] no items to output this cycle I1121 12:41:36.413411 3798194 cpu.go:275] no items to output this cycle E1121 12:41:46.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:41:46.412260 3798194 memory.go:184] no items to output this cycle I1121 12:41:46.412405 3798194 cpu.go:275] no items to output this cycle E1121 12:41:56.412273 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:41:56.412307 3798194 memory.go:191] Add success. W1121 12:41:56.412351 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:41:56.412364 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:41:56.412367 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:41:56.416595 3798194 cpu.go:282] Add success. I1121 12:41:56.441133 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:41:56.441654 3798194 net.go:642] Add success. I1121 12:41:56.443324 3798194 net.go:778] primary dev: ETH0 I1121 12:41:56.443348 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:41:56.443369 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:41:57.130151 3798194 disk_info.go:125] begin check local disk info of client I1121 12:41:57.133674 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:41:57.133683 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004f1600 0xc0004f1640] I1121 12:41:57.450206 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:41:57.450381 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:41:57.450436 3798194 disk_worker.go:723] disk space is not compliant W1121 12:41:57.450439 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:41:57.458496 3798194 disk_worker.go:509] system disk:vda1 I1121 12:41:57.458540 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:41:58.446428 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:41:59.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:41:59.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:41:59.450496 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:41:59.471049 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:42:06.411232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:42:06.411256 3798194 memory.go:184] no items to output this cycle I1121 12:42:06.411403 3798194 cpu.go:275] no items to output this cycle E1121 12:42:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:42:16.412259 3798194 memory.go:184] no items to output this cycle I1121 12:42:16.412409 3798194 cpu.go:275] no items to output this cycle I1121 12:42:17.466990 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:42:17.467001 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:42:26.411132 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:42:26.411167 3798194 memory.go:191] Add success. I1121 12:42:26.411481 3798194 cpu.go:282] Add success. I1121 12:42:26.441282 3798194 net.go:642] Add success. I1121 12:42:26.445541 3798194 net.go:778] primary dev: ETH0 I1121 12:42:26.445565 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:42:26.445580 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:42:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:42:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:42:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:42:36.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:42:36.413256 3798194 memory.go:184] no items to output this cycle I1121 12:42:36.413398 3798194 cpu.go:275] no items to output this cycle E1121 12:42:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:42:46.411985 3798194 memory.go:184] no items to output this cycle I1121 12:42:46.412150 3798194 cpu.go:275] no items to output this cycle E1121 12:42:56.412484 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:42:56.412517 3798194 memory.go:191] Add success. W1121 12:42:56.412541 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:42:56.412552 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:42:56.412555 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:42:56.416756 3798194 cpu.go:282] Add success. I1121 12:42:56.441140 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:42:56.441624 3798194 net.go:642] Add success. I1121 12:42:56.443181 3798194 net.go:778] primary dev: ETH0 I1121 12:42:56.443199 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:42:56.443211 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:42:57.133779 3798194 disk_info.go:125] begin check local disk info of client I1121 12:42:57.137285 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:42:57.137294 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003da680 0xc0003da6c0] I1121 12:42:57.450262 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:42:57.450417 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:42:57.450472 3798194 disk_worker.go:723] disk space is not compliant W1121 12:42:57.450475 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:42:57.458536 3798194 disk_worker.go:509] system disk:vda1 I1121 12:42:57.458578 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:42:58.446375 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:42:59.450477 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:42:59.450555 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:42:59.450579 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:42:59.471090 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:43:06.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:43:06.411987 3798194 memory.go:184] no items to output this cycle I1121 12:43:06.412136 3798194 cpu.go:275] no items to output this cycle E1121 12:43:16.411971 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:43:16.411994 3798194 memory.go:184] no items to output this cycle I1121 12:43:16.412155 3798194 cpu.go:275] no items to output this cycle E1121 12:43:26.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:43:26.413282 3798194 memory.go:191] Add success. I1121 12:43:26.417824 3798194 cpu.go:282] Add success. I1121 12:43:26.441323 3798194 net.go:642] Add success. I1121 12:43:26.442880 3798194 net.go:778] primary dev: ETH0 I1121 12:43:26.442895 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:43:26.442910 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:43:29.450203 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:43:29.450293 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:43:29.450324 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:43:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:43:36.411975 3798194 memory.go:184] no items to output this cycle I1121 12:43:36.412121 3798194 cpu.go:275] no items to output this cycle E1121 12:43:46.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:43:46.412886 3798194 memory.go:184] no items to output this cycle I1121 12:43:46.413033 3798194 cpu.go:275] no items to output this cycle E1121 12:43:56.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:43:56.413277 3798194 memory.go:191] Add success. W1121 12:43:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:43:56.413322 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:43:56.413326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:43:56.418398 3798194 cpu.go:282] Add success. I1121 12:43:56.441884 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:43:56.442498 3798194 net.go:642] Add success. I1121 12:43:56.444266 3798194 net.go:778] primary dev: ETH0 I1121 12:43:56.444281 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:43:56.444292 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:43:57.140145 3798194 disk_info.go:125] begin check local disk info of client I1121 12:43:57.143728 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:43:57.143737 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002eba80 0xc0002ebac0] I1121 12:43:57.445236 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:43:57.445454 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:43:57.445519 3798194 disk_worker.go:723] disk space is not compliant W1121 12:43:57.445523 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:43:57.448000 3798194 disk_worker.go:509] system disk:vda1 I1121 12:43:57.448038 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:43:58.445987 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:43:59.450030 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:43:59.450134 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:43:59.450161 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:43:59.471648 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:44:06.412250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:44:06.412272 3798194 memory.go:184] no items to output this cycle I1121 12:44:06.412420 3798194 cpu.go:275] no items to output this cycle E1121 12:44:16.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:44:16.412253 3798194 memory.go:184] no items to output this cycle I1121 12:44:16.412397 3798194 cpu.go:275] no items to output this cycle E1121 12:44:26.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:44:26.412888 3798194 memory.go:191] Add success. I1121 12:44:26.417169 3798194 cpu.go:282] Add success. I1121 12:44:26.441691 3798194 net.go:642] Add success. I1121 12:44:26.443234 3798194 net.go:778] primary dev: ETH0 I1121 12:44:26.443249 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:44:26.443262 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:44:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:44:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:44:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:44:36.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:44:36.412888 3798194 memory.go:184] no items to output this cycle I1121 12:44:36.413058 3798194 cpu.go:275] no items to output this cycle E1121 12:44:46.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:44:46.412257 3798194 memory.go:184] no items to output this cycle I1121 12:44:46.412408 3798194 cpu.go:275] no items to output this cycle E1121 12:44:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:44:56.411961 3798194 memory.go:191] Add success. W1121 12:44:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:44:56.412007 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:44:56.412010 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:44:56.412381 3798194 cpu.go:282] Add success. I1121 12:44:56.441824 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:44:56.442344 3798194 net.go:642] Add success. I1121 12:44:56.446355 3798194 net.go:778] primary dev: ETH0 I1121 12:44:56.446369 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:44:56.446382 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:44:57.143841 3798194 disk_info.go:125] begin check local disk info of client I1121 12:44:57.147389 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:44:57.147398 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030f3c0 0xc00030f400] I1121 12:44:57.450639 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:44:57.450841 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:44:57.450851 3798194 disk_worker.go:723] disk space is not compliant W1121 12:44:57.450854 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:44:57.459011 3798194 disk_worker.go:509] system disk:vda1 I1121 12:44:57.459054 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:44:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:44:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:44:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:44:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:44:59.471656 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:45:06.413008 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:45:06.413031 3798194 memory.go:184] no items to output this cycle I1121 12:45:06.413210 3798194 cpu.go:275] no items to output this cycle E1121 12:45:16.412315 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:45:16.412343 3798194 memory.go:184] no items to output this cycle I1121 12:45:16.412576 3798194 cpu.go:275] no items to output this cycle I1121 12:45:17.467145 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:45:17.467154 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:45:26.411197 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:45:26.411230 3798194 memory.go:191] Add success. I1121 12:45:26.411579 3798194 cpu.go:282] Add success. I1121 12:45:26.441280 3798194 net.go:642] Add success. I1121 12:45:26.445210 3798194 net.go:778] primary dev: ETH0 I1121 12:45:26.445226 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:45:26.445238 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:45:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:45:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:45:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:45:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:45:36.412266 3798194 memory.go:184] no items to output this cycle I1121 12:45:36.412415 3798194 cpu.go:275] no items to output this cycle E1121 12:45:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:45:46.411976 3798194 memory.go:184] no items to output this cycle I1121 12:45:46.412137 3798194 cpu.go:275] no items to output this cycle E1121 12:45:56.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:45:56.413258 3798194 memory.go:191] Add success. W1121 12:45:56.413291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:45:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:45:56.413307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:45:56.417680 3798194 cpu.go:282] Add success. I1121 12:45:56.443501 3798194 event_worker.go:152] Polling the log file for events... I1121 12:45:56.443717 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:45:56.444211 3798194 net.go:642] Add success. I1121 12:45:56.445812 3798194 net.go:778] primary dev: ETH0 I1121 12:45:56.445827 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:45:56.445845 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:45:57.150051 3798194 disk_info.go:125] begin check local disk info of client I1121 12:45:57.153615 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:45:57.153624 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041ef40 0xc00041ef80] E1121 12:45:57.450585 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 12:45:57.450599 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:45:57.450761 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:45:57.450814 3798194 disk_worker.go:723] disk space is not compliant W1121 12:45:57.450817 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:45:57.459444 3798194 disk_worker.go:509] system disk:vda1 I1121 12:45:57.459487 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 12:45:58.447872 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 12:45:58.447888 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 12:45:59.448648 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 12:45:59.449725 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:45:59.449779 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:45:59.449799 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:45:59.471380 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:46:06.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:46:06.411974 3798194 memory.go:184] no items to output this cycle I1121 12:46:06.412130 3798194 cpu.go:275] no items to output this cycle E1121 12:46:16.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:46:16.411991 3798194 memory.go:184] no items to output this cycle I1121 12:46:16.412147 3798194 cpu.go:275] no items to output this cycle E1121 12:46:26.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:46:26.411948 3798194 memory.go:191] Add success. I1121 12:46:26.412298 3798194 cpu.go:282] Add success. I1121 12:46:26.441755 3798194 net.go:642] Add success. I1121 12:46:26.446399 3798194 net.go:778] primary dev: ETH0 I1121 12:46:26.446416 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:46:26.446431 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:46:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:46:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:46:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:46:36.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:46:36.413254 3798194 memory.go:184] no items to output this cycle I1121 12:46:36.413397 3798194 cpu.go:275] no items to output this cycle E1121 12:46:46.412874 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:46:46.412896 3798194 memory.go:184] no items to output this cycle I1121 12:46:46.413045 3798194 cpu.go:275] no items to output this cycle E1121 12:46:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:46:56.411955 3798194 memory.go:191] Add success. W1121 12:46:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:46:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:46:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:46:56.412351 3798194 cpu.go:282] Add success. I1121 12:46:56.441810 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:46:56.442407 3798194 net.go:642] Add success. I1121 12:46:56.446582 3798194 net.go:778] primary dev: ETH0 I1121 12:46:56.446596 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:46:56.446608 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:46:57.153743 3798194 disk_info.go:125] begin check local disk info of client I1121 12:46:57.157267 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:46:57.157275 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037db80 0xc00037dbc0] I1121 12:46:57.450572 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:46:57.450726 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:46:57.450778 3798194 disk_worker.go:723] disk space is not compliant W1121 12:46:57.450781 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:46:57.458900 3798194 disk_worker.go:509] system disk:vda1 I1121 12:46:57.458940 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:46:58.446196 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:46:59.450301 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:46:59.450379 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:46:59.450404 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:46:59.470933 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:47:06.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:47:06.412262 3798194 memory.go:184] no items to output this cycle I1121 12:47:06.412410 3798194 cpu.go:275] no items to output this cycle E1121 12:47:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:47:16.412261 3798194 memory.go:184] no items to output this cycle I1121 12:47:16.412406 3798194 cpu.go:275] no items to output this cycle E1121 12:47:26.411930 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:47:26.411970 3798194 memory.go:191] Add success. I1121 12:47:26.412330 3798194 cpu.go:282] Add success. I1121 12:47:26.441977 3798194 net.go:642] Add success. I1121 12:47:26.446144 3798194 net.go:778] primary dev: ETH0 I1121 12:47:26.446159 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:47:26.446173 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:47:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:47:29.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:47:29.450145 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:47:36.412779 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:47:36.412801 3798194 memory.go:184] no items to output this cycle I1121 12:47:36.413167 3798194 cpu.go:275] no items to output this cycle E1121 12:47:46.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:47:46.412255 3798194 memory.go:184] no items to output this cycle I1121 12:47:46.412402 3798194 cpu.go:275] no items to output this cycle E1121 12:47:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:47:56.411958 3798194 memory.go:191] Add success. W1121 12:47:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:47:56.412011 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:47:56.412014 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:47:56.412374 3798194 cpu.go:282] Add success. I1121 12:47:56.441823 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:47:56.442323 3798194 net.go:642] Add success. I1121 12:47:56.446508 3798194 net.go:778] primary dev: ETH0 I1121 12:47:56.446522 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:47:56.446534 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:47:57.157391 3798194 disk_info.go:125] begin check local disk info of client I1121 12:47:57.160943 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:47:57.160952 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037ca40 0xc00037ca80] I1121 12:47:57.450332 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:47:57.450545 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:47:57.450556 3798194 disk_worker.go:723] disk space is not compliant W1121 12:47:57.450559 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:47:57.458690 3798194 disk_worker.go:509] system disk:vda1 I1121 12:47:57.458731 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:47:58.446381 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:47:59.450004 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:47:59.450074 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:47:59.450100 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:47:59.471627 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:48:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:48:06.411978 3798194 memory.go:184] no items to output this cycle I1121 12:48:06.412130 3798194 cpu.go:275] no items to output this cycle E1121 12:48:16.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:48:16.411966 3798194 memory.go:184] no items to output this cycle I1121 12:48:16.412131 3798194 cpu.go:275] no items to output this cycle I1121 12:48:17.467307 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:48:17.467317 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:48:26.411278 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:48:26.411309 3798194 memory.go:191] Add success. I1121 12:48:26.411654 3798194 cpu.go:282] Add success. I1121 12:48:26.441297 3798194 net.go:642] Add success. I1121 12:48:26.445124 3798194 net.go:778] primary dev: ETH0 I1121 12:48:26.445139 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:48:26.445152 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:48:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:48:29.450086 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:48:29.450123 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:48:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:48:36.411979 3798194 memory.go:184] no items to output this cycle I1121 12:48:36.412131 3798194 cpu.go:275] no items to output this cycle E1121 12:48:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:48:46.411977 3798194 memory.go:184] no items to output this cycle I1121 12:48:46.412137 3798194 cpu.go:275] no items to output this cycle E1121 12:48:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:48:56.412269 3798194 memory.go:191] Add success. W1121 12:48:56.412301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:48:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:48:56.412316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:48:56.416683 3798194 cpu.go:282] Add success. I1121 12:48:56.440944 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:48:56.441534 3798194 net.go:642] Add success. I1121 12:48:56.443063 3798194 net.go:778] primary dev: ETH0 I1121 12:48:56.443077 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:48:56.443091 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:48:57.164320 3798194 disk_info.go:125] begin check local disk info of client I1121 12:48:57.167981 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:48:57.167991 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00049eb00 0xc00049eb40] I1121 12:48:57.450618 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:48:57.450829 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:48:57.450839 3798194 disk_worker.go:723] disk space is not compliant W1121 12:48:57.450842 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:48:57.459756 3798194 disk_worker.go:509] system disk:vda1 I1121 12:48:57.459796 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:48:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:48:59.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:48:59.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:48:59.450511 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:48:59.471020 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:49:06.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:49:06.413251 3798194 memory.go:184] no items to output this cycle I1121 12:49:06.413395 3798194 cpu.go:275] no items to output this cycle E1121 12:49:16.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:49:16.411993 3798194 memory.go:184] no items to output this cycle I1121 12:49:16.412151 3798194 cpu.go:275] no items to output this cycle E1121 12:49:26.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:49:26.411960 3798194 memory.go:191] Add success. I1121 12:49:26.412294 3798194 cpu.go:282] Add success. I1121 12:49:26.441896 3798194 net.go:642] Add success. I1121 12:49:26.446090 3798194 net.go:778] primary dev: ETH0 I1121 12:49:26.446126 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:49:26.446146 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:49:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:49:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:49:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:49:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:49:36.412259 3798194 memory.go:184] no items to output this cycle I1121 12:49:36.412408 3798194 cpu.go:275] no items to output this cycle E1121 12:49:46.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:49:46.412255 3798194 memory.go:184] no items to output this cycle I1121 12:49:46.412403 3798194 cpu.go:275] no items to output this cycle E1121 12:49:56.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:49:56.412263 3798194 memory.go:191] Add success. W1121 12:49:56.412294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:49:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:49:56.412309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:49:56.419600 3798194 cpu.go:282] Add success. I1121 12:49:56.441121 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:49:56.441596 3798194 net.go:642] Add success. I1121 12:49:56.443145 3798194 net.go:778] primary dev: ETH0 I1121 12:49:56.443159 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:49:56.443171 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:49:57.168096 3798194 disk_info.go:125] begin check local disk info of client I1121 12:49:57.171660 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:49:57.171671 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b3100 0xc0003b3140] I1121 12:49:57.450236 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:49:57.450401 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:49:57.450457 3798194 disk_worker.go:723] disk space is not compliant W1121 12:49:57.450460 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:49:57.458722 3798194 disk_worker.go:509] system disk:vda1 I1121 12:49:57.458764 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:49:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:49:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:49:59.450467 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:49:59.450491 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:49:59.470992 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:50:06.412821 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:50:06.412842 3798194 memory.go:184] no items to output this cycle I1121 12:50:06.412981 3798194 cpu.go:275] no items to output this cycle E1121 12:50:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:50:16.411977 3798194 memory.go:184] no items to output this cycle I1121 12:50:16.412136 3798194 cpu.go:275] no items to output this cycle E1121 12:50:26.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:50:26.412264 3798194 memory.go:191] Add success. I1121 12:50:26.416554 3798194 cpu.go:282] Add success. I1121 12:50:26.441311 3798194 net.go:642] Add success. I1121 12:50:26.442775 3798194 net.go:778] primary dev: ETH0 I1121 12:50:26.442789 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:50:26.442803 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:50:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:50:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:50:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:50:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:50:36.412258 3798194 memory.go:184] no items to output this cycle I1121 12:50:36.412411 3798194 cpu.go:275] no items to output this cycle E1121 12:50:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:50:46.412259 3798194 memory.go:184] no items to output this cycle I1121 12:50:46.412401 3798194 cpu.go:275] no items to output this cycle E1121 12:50:56.413250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:50:56.413291 3798194 memory.go:191] Add success. W1121 12:50:56.413325 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:50:56.413337 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:50:56.413340 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:50:56.418330 3798194 cpu.go:282] Add success. I1121 12:50:56.442989 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:50:56.443593 3798194 net.go:642] Add success. I1121 12:50:56.449085 3798194 net.go:778] primary dev: ETH0 I1121 12:50:56.449100 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:50:56.449126 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:50:57.175152 3798194 disk_info.go:125] begin check local disk info of client I1121 12:50:57.178720 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:50:57.178728 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002744c0 0xc000274500] E1121 12:50:57.445299 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 12:50:57.445318 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:50:57.445533 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:50:57.445597 3798194 disk_worker.go:723] disk space is not compliant W1121 12:50:57.445601 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:50:57.448090 3798194 disk_worker.go:509] system disk:vda1 I1121 12:50:57.448154 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 12:50:58.448192 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 12:50:58.448207 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 12:50:59.448878 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 12:50:59.449954 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:50:59.450007 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:50:59.450026 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:50:59.471564 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:51:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:51:06.412261 3798194 memory.go:184] no items to output this cycle I1121 12:51:06.412408 3798194 cpu.go:275] no items to output this cycle E1121 12:51:16.412815 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:51:16.412844 3798194 memory.go:184] no items to output this cycle I1121 12:51:16.413049 3798194 cpu.go:275] no items to output this cycle I1121 12:51:17.467476 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:51:17.467488 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:51:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:51:26.413266 3798194 memory.go:191] Add success. I1121 12:51:26.417751 3798194 cpu.go:282] Add success. I1121 12:51:26.441295 3798194 net.go:642] Add success. I1121 12:51:26.442687 3798194 net.go:778] primary dev: ETH0 I1121 12:51:26.442701 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:51:26.442714 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:51:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:51:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:51:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:51:36.413317 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:51:36.413337 3798194 memory.go:184] no items to output this cycle I1121 12:51:36.413482 3798194 cpu.go:275] no items to output this cycle E1121 12:51:46.413432 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:51:46.413454 3798194 memory.go:184] no items to output this cycle I1121 12:51:46.413605 3798194 cpu.go:275] no items to output this cycle E1121 12:51:56.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:51:56.412272 3798194 memory.go:191] Add success. W1121 12:51:56.412301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:51:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:51:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:51:56.416634 3798194 cpu.go:282] Add success. I1121 12:51:56.441158 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:51:56.442283 3798194 net.go:642] Add success. I1121 12:51:56.443821 3798194 net.go:778] primary dev: ETH0 I1121 12:51:56.443836 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:51:56.443848 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:51:57.178831 3798194 disk_info.go:125] begin check local disk info of client I1121 12:51:57.182301 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:51:57.182309 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0005111c0 0xc000511200] I1121 12:51:57.450654 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:51:57.450814 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:51:57.450867 3798194 disk_worker.go:723] disk space is not compliant W1121 12:51:57.450870 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:51:57.458957 3798194 disk_worker.go:509] system disk:vda1 I1121 12:51:57.459000 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:51:58.446383 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:51:59.450486 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:51:59.450564 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:51:59.450589 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:51:59.471101 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:52:06.412251 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:52:06.412273 3798194 memory.go:184] no items to output this cycle I1121 12:52:06.412429 3798194 cpu.go:275] no items to output this cycle E1121 12:52:16.413253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:52:16.413276 3798194 memory.go:184] no items to output this cycle I1121 12:52:16.413441 3798194 cpu.go:275] no items to output this cycle E1121 12:52:26.411942 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:52:26.411977 3798194 memory.go:191] Add success. I1121 12:52:26.412347 3798194 cpu.go:282] Add success. I1121 12:52:26.442007 3798194 net.go:642] Add success. I1121 12:52:26.445656 3798194 net.go:778] primary dev: ETH0 I1121 12:52:26.445672 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:52:26.445684 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:52:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:52:29.450119 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:52:29.450148 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:52:36.412085 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:52:36.412106 3798194 memory.go:184] no items to output this cycle I1121 12:52:36.412269 3798194 cpu.go:275] no items to output this cycle E1121 12:52:46.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:52:46.412883 3798194 memory.go:184] no items to output this cycle I1121 12:52:46.413040 3798194 cpu.go:275] no items to output this cycle E1121 12:52:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:52:56.411957 3798194 memory.go:191] Add success. W1121 12:52:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:52:56.412006 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:52:56.412010 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:52:56.412358 3798194 cpu.go:282] Add success. I1121 12:52:56.441805 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:52:56.442322 3798194 net.go:642] Add success. I1121 12:52:56.446205 3798194 net.go:778] primary dev: ETH0 I1121 12:52:56.446220 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:52:56.446233 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:52:57.182406 3798194 disk_info.go:125] begin check local disk info of client I1121 12:52:57.185988 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:52:57.185997 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00034ef80 0xc00034efc0] I1121 12:52:57.451813 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:52:57.451973 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:52:57.452029 3798194 disk_worker.go:723] disk space is not compliant W1121 12:52:57.452032 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:52:57.463900 3798194 disk_worker.go:509] system disk:vda1 I1121 12:52:57.463952 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:52:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:52:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:52:59.450462 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:52:59.450488 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:52:59.470996 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:53:06.412089 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:53:06.412126 3798194 memory.go:184] no items to output this cycle I1121 12:53:06.412402 3798194 cpu.go:275] no items to output this cycle E1121 12:53:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:53:16.411973 3798194 memory.go:184] no items to output this cycle I1121 12:53:16.412137 3798194 cpu.go:275] no items to output this cycle E1121 12:53:26.413255 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:53:26.413293 3798194 memory.go:191] Add success. I1121 12:53:26.417921 3798194 cpu.go:282] Add success. I1121 12:53:26.441528 3798194 net.go:642] Add success. I1121 12:53:26.442986 3798194 net.go:778] primary dev: ETH0 I1121 12:53:26.443001 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:53:26.443015 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:53:29.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:53:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:53:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:53:36.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:53:36.413262 3798194 memory.go:184] no items to output this cycle I1121 12:53:36.413412 3798194 cpu.go:275] no items to output this cycle E1121 12:53:46.411971 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:53:46.411993 3798194 memory.go:184] no items to output this cycle I1121 12:53:46.412158 3798194 cpu.go:275] no items to output this cycle E1121 12:53:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:53:56.411953 3798194 memory.go:191] Add success. W1121 12:53:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:53:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:53:56.412002 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:53:56.412335 3798194 cpu.go:282] Add success. I1121 12:53:56.441807 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:53:56.442321 3798194 net.go:642] Add success. I1121 12:53:56.446776 3798194 net.go:778] primary dev: ETH0 I1121 12:53:56.446791 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:53:56.446813 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:53:57.186104 3798194 disk_info.go:125] begin check local disk info of client I1121 12:53:57.189733 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:53:57.189742 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000353a40 0xc000353a80] I1121 12:53:57.450324 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:53:57.450489 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:53:57.450543 3798194 disk_worker.go:723] disk space is not compliant W1121 12:53:57.450546 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:53:57.458730 3798194 disk_worker.go:509] system disk:vda1 I1121 12:53:57.458783 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:53:58.446010 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:53:59.450403 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:53:59.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:53:59.450508 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:53:59.471021 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:54:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:54:06.412261 3798194 memory.go:184] no items to output this cycle I1121 12:54:06.412409 3798194 cpu.go:275] no items to output this cycle E1121 12:54:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:54:16.411973 3798194 memory.go:184] no items to output this cycle I1121 12:54:16.412126 3798194 cpu.go:275] no items to output this cycle I1121 12:54:17.468185 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:54:17.468194 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:54:26.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:54:26.412278 3798194 memory.go:191] Add success. I1121 12:54:26.416798 3798194 cpu.go:282] Add success. I1121 12:54:26.441608 3798194 net.go:642] Add success. I1121 12:54:26.443043 3798194 net.go:778] primary dev: ETH0 I1121 12:54:26.443060 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:54:26.443075 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:54:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:54:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:54:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:54:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:54:36.412253 3798194 memory.go:184] no items to output this cycle I1121 12:54:36.412394 3798194 cpu.go:275] no items to output this cycle E1121 12:54:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:54:46.412256 3798194 memory.go:184] no items to output this cycle I1121 12:54:46.412397 3798194 cpu.go:275] no items to output this cycle E1121 12:54:56.413301 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:54:56.413334 3798194 memory.go:191] Add success. W1121 12:54:56.413369 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:54:56.413380 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:54:56.413384 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:54:56.418203 3798194 cpu.go:282] Add success. I1121 12:54:56.441722 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:54:56.442200 3798194 net.go:642] Add success. I1121 12:54:56.443740 3798194 net.go:778] primary dev: ETH0 I1121 12:54:56.443755 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:54:56.443767 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:54:57.189845 3798194 disk_info.go:125] begin check local disk info of client I1121 12:54:57.193484 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:54:57.193493 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000353440 0xc000353480] I1121 12:54:57.445342 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:54:57.445616 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:54:57.445629 3798194 disk_worker.go:723] disk space is not compliant W1121 12:54:57.445632 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:54:57.448271 3798194 disk_worker.go:509] system disk:vda1 I1121 12:54:57.448313 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:54:58.445986 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:54:59.450038 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:54:59.450136 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:54:59.450162 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:54:59.471648 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:55:06.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:55:06.411972 3798194 memory.go:184] no items to output this cycle I1121 12:55:06.412134 3798194 cpu.go:275] no items to output this cycle E1121 12:55:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:55:16.412256 3798194 memory.go:184] no items to output this cycle I1121 12:55:16.412404 3798194 cpu.go:275] no items to output this cycle E1121 12:55:26.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:55:26.411961 3798194 memory.go:191] Add success. I1121 12:55:26.412315 3798194 cpu.go:282] Add success. I1121 12:55:26.442096 3798194 net.go:642] Add success. I1121 12:55:26.446805 3798194 net.go:778] primary dev: ETH0 I1121 12:55:26.446820 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:55:26.446833 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:55:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:55:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:55:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:55:36.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:55:36.411982 3798194 memory.go:184] no items to output this cycle I1121 12:55:36.412131 3798194 cpu.go:275] no items to output this cycle E1121 12:55:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:55:46.411976 3798194 memory.go:184] no items to output this cycle I1121 12:55:46.412143 3798194 cpu.go:275] no items to output this cycle E1121 12:55:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:55:56.411952 3798194 memory.go:191] Add success. W1121 12:55:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:55:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:55:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:55:56.412727 3798194 cpu.go:282] Add success. I1121 12:55:56.446185 3798194 event_worker.go:152] Polling the log file for events... I1121 12:55:56.446413 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:55:56.447030 3798194 net.go:642] Add success. I1121 12:55:56.451336 3798194 net.go:778] primary dev: ETH0 I1121 12:55:56.451351 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:55:56.451364 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:55:57.196145 3798194 disk_info.go:125] begin check local disk info of client I1121 12:55:57.199680 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:55:57.199689 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071480 0xc0000714c0] E1121 12:55:57.450395 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 12:55:57.450410 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:55:57.450571 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:55:57.450639 3798194 disk_worker.go:723] disk space is not compliant W1121 12:55:57.450643 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:55:57.459907 3798194 disk_worker.go:509] system disk:vda1 I1121 12:55:57.459949 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 12:55:58.447808 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 12:55:58.447822 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 12:55:59.448978 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 12:55:59.450054 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:55:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:55:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:55:59.471640 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:56:06.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:56:06.413260 3798194 memory.go:184] no items to output this cycle I1121 12:56:06.413402 3798194 cpu.go:275] no items to output this cycle E1121 12:56:16.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:56:16.412882 3798194 memory.go:184] no items to output this cycle I1121 12:56:16.413032 3798194 cpu.go:275] no items to output this cycle E1121 12:56:26.411937 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:56:26.411970 3798194 memory.go:191] Add success. I1121 12:56:26.412309 3798194 cpu.go:282] Add success. I1121 12:56:26.442093 3798194 net.go:642] Add success. I1121 12:56:26.445907 3798194 net.go:778] primary dev: ETH0 I1121 12:56:26.445922 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:56:26.445933 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:56:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:56:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:56:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:56:36.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:56:36.411975 3798194 memory.go:184] no items to output this cycle I1121 12:56:36.412135 3798194 cpu.go:275] no items to output this cycle E1121 12:56:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:56:46.411980 3798194 memory.go:184] no items to output this cycle I1121 12:56:46.412133 3798194 cpu.go:275] no items to output this cycle E1121 12:56:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:56:56.412269 3798194 memory.go:191] Add success. W1121 12:56:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:56:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:56:56.412313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:56:56.416787 3798194 cpu.go:282] Add success. I1121 12:56:56.441342 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:56:56.441822 3798194 net.go:642] Add success. I1121 12:56:56.443306 3798194 net.go:778] primary dev: ETH0 I1121 12:56:56.443322 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:56:56.443337 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:56:57.202138 3798194 disk_info.go:125] begin check local disk info of client I1121 12:56:57.207298 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:56:57.207306 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000426500 0xc000426540] I1121 12:56:57.450255 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:56:57.450469 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:56:57.450480 3798194 disk_worker.go:723] disk space is not compliant W1121 12:56:57.450483 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:56:57.460316 3798194 disk_worker.go:509] system disk:vda1 I1121 12:56:57.460354 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:56:58.446009 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:56:59.453143 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:56:59.453215 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:56:59.453242 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:56:59.470687 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:57:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:57:06.411985 3798194 memory.go:184] no items to output this cycle I1121 12:57:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 12:57:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:57:16.413252 3798194 memory.go:184] no items to output this cycle I1121 12:57:16.413398 3798194 cpu.go:275] no items to output this cycle I1121 12:57:17.468352 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 12:57:17.468360 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 12:57:26.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:57:26.412898 3798194 memory.go:191] Add success. I1121 12:57:26.417072 3798194 cpu.go:282] Add success. I1121 12:57:26.441767 3798194 net.go:642] Add success. I1121 12:57:26.443180 3798194 net.go:778] primary dev: ETH0 I1121 12:57:26.443194 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:57:26.443207 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:57:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:57:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:57:29.450150 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:57:36.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:57:36.413256 3798194 memory.go:184] no items to output this cycle I1121 12:57:36.413401 3798194 cpu.go:275] no items to output this cycle E1121 12:57:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:57:46.411975 3798194 memory.go:184] no items to output this cycle I1121 12:57:46.412147 3798194 cpu.go:275] no items to output this cycle E1121 12:57:56.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:57:56.412889 3798194 memory.go:191] Add success. W1121 12:57:56.412926 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:57:56.412938 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:57:56.412941 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:57:56.417245 3798194 cpu.go:282] Add success. I1121 12:57:56.441636 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:57:56.442104 3798194 net.go:642] Add success. I1121 12:57:56.443583 3798194 net.go:778] primary dev: ETH0 I1121 12:57:56.443598 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:57:56.443610 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:57:57.207410 3798194 disk_info.go:125] begin check local disk info of client I1121 12:57:57.211002 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:57:57.211011 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000356880 0xc0003568c0] I1121 12:57:57.450251 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:57:57.450414 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:57:57.450480 3798194 disk_worker.go:723] disk space is not compliant W1121 12:57:57.450484 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:57:57.458569 3798194 disk_worker.go:509] system disk:vda1 I1121 12:57:57.458609 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:57:58.446492 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:57:59.450603 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:57:59.450687 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:57:59.450712 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:57:59.471221 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:58:06.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:58:06.412879 3798194 memory.go:184] no items to output this cycle I1121 12:58:06.413029 3798194 cpu.go:275] no items to output this cycle E1121 12:58:16.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:58:16.412270 3798194 memory.go:184] no items to output this cycle I1121 12:58:16.412435 3798194 cpu.go:275] no items to output this cycle E1121 12:58:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:58:26.413263 3798194 memory.go:191] Add success. I1121 12:58:26.417912 3798194 cpu.go:282] Add success. I1121 12:58:26.442237 3798194 net.go:642] Add success. I1121 12:58:26.443882 3798194 net.go:778] primary dev: ETH0 I1121 12:58:26.443898 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:58:26.443911 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:58:29.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:58:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:58:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:58:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:58:36.411976 3798194 memory.go:184] no items to output this cycle I1121 12:58:36.412133 3798194 cpu.go:275] no items to output this cycle E1121 12:58:46.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:58:46.411985 3798194 memory.go:184] no items to output this cycle I1121 12:58:46.412147 3798194 cpu.go:275] no items to output this cycle E1121 12:58:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:58:56.411954 3798194 memory.go:191] Add success. W1121 12:58:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:58:56.412009 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:58:56.412013 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:58:56.412359 3798194 cpu.go:282] Add success. I1121 12:58:56.441803 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:58:56.442409 3798194 net.go:642] Add success. I1121 12:58:56.447036 3798194 net.go:778] primary dev: ETH0 I1121 12:58:56.447052 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:58:56.447064 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:58:57.214547 3798194 disk_info.go:125] begin check local disk info of client I1121 12:58:57.221381 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:58:57.221389 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000274e80 0xc000274ec0] I1121 12:58:57.451342 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:58:57.451518 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:58:57.451573 3798194 disk_worker.go:723] disk space is not compliant W1121 12:58:57.451576 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:58:57.466103 3798194 disk_worker.go:509] system disk:vda1 I1121 12:58:57.466166 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:58:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:58:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:58:59.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:58:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:58:59.471634 3798194 disk_local_worker.go:436] Get disk info: [] E1121 12:59:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:59:06.412253 3798194 memory.go:184] no items to output this cycle I1121 12:59:06.412406 3798194 cpu.go:275] no items to output this cycle E1121 12:59:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:59:16.411979 3798194 memory.go:184] no items to output this cycle I1121 12:59:16.412139 3798194 cpu.go:275] no items to output this cycle E1121 12:59:26.411242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:59:26.411276 3798194 memory.go:191] Add success. I1121 12:59:26.415675 3798194 cpu.go:282] Add success. I1121 12:59:26.442246 3798194 net.go:642] Add success. I1121 12:59:26.443701 3798194 net.go:778] primary dev: ETH0 I1121 12:59:26.443716 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:59:26.443728 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:59:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:59:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:59:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 12:59:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:59:36.413254 3798194 memory.go:184] no items to output this cycle I1121 12:59:36.413394 3798194 cpu.go:275] no items to output this cycle E1121 12:59:46.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:59:46.411983 3798194 memory.go:184] no items to output this cycle I1121 12:59:46.412149 3798194 cpu.go:275] no items to output this cycle E1121 12:59:56.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 12:59:56.413260 3798194 memory.go:191] Add success. W1121 12:59:56.413287 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 12:59:56.413299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 12:59:56.413302 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 12:59:56.417897 3798194 cpu.go:282] Add success. I1121 12:59:56.440965 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 12:59:56.441562 3798194 net.go:642] Add success. I1121 12:59:56.443320 3798194 net.go:778] primary dev: ETH0 I1121 12:59:56.443334 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 12:59:56.443347 3798194 net.go:659] Add eni metrics success, count: 7 I1121 12:59:57.221495 3798194 disk_info.go:125] begin check local disk info of client I1121 12:59:57.225075 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 12:59:57.225084 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000475a80 0xc000475ac0] I1121 12:59:57.445227 3798194 custom_config.go:106] query custom config with name: gpu W1121 12:59:57.445496 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 12:59:57.445508 3798194 disk_worker.go:723] disk space is not compliant W1121 12:59:57.445512 3798194 disk_worker.go:743] disk inode is not compliant I1121 12:59:57.448019 3798194 disk_worker.go:509] system disk:vda1 I1121 12:59:57.448058 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 12:59:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 12:59:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 12:59:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 12:59:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 12:59:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:00:06.411240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:00:06.411261 3798194 memory.go:184] no items to output this cycle I1121 13:00:06.411403 3798194 cpu.go:275] no items to output this cycle E1121 13:00:16.412844 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:00:16.412869 3798194 memory.go:184] no items to output this cycle I1121 13:00:16.413021 3798194 cpu.go:275] no items to output this cycle I1121 13:00:17.468524 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:00:17.468533 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:00:26.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:00:26.413264 3798194 memory.go:191] Add success. I1121 13:00:26.417359 3798194 cpu.go:282] Add success. I1121 13:00:26.441869 3798194 net.go:642] Add success. I1121 13:00:26.443239 3798194 net.go:778] primary dev: ETH0 I1121 13:00:26.443255 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:00:26.443268 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:00:29.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:00:29.450490 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:00:29.450516 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:00:36.411987 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:00:36.412008 3798194 memory.go:184] no items to output this cycle I1121 13:00:36.412169 3798194 cpu.go:275] no items to output this cycle E1121 13:00:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:00:46.412259 3798194 memory.go:184] no items to output this cycle I1121 13:00:46.412410 3798194 cpu.go:275] no items to output this cycle E1121 13:00:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:00:56.411953 3798194 memory.go:191] Add success. W1121 13:00:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:00:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:00:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:00:56.412330 3798194 cpu.go:282] Add success. I1121 13:00:56.445257 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:00:56.445741 3798194 net.go:642] Add success. I1121 13:00:56.450587 3798194 net.go:778] primary dev: ETH0 I1121 13:00:56.450602 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:00:56.450616 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:00:57.225156 3798194 disk_info.go:125] begin check local disk info of client I1121 13:00:57.228815 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:00:57.228824 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f8300 0xc0003f8340] E1121 13:00:57.445829 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 13:00:57.445848 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:00:57.446103 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:00:57.446124 3798194 disk_worker.go:723] disk space is not compliant W1121 13:00:57.446128 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:00:57.448579 3798194 disk_worker.go:509] system disk:vda1 I1121 13:00:57.448619 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 13:00:58.448172 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 13:00:58.448187 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 13:00:59.448605 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 13:00:59.449680 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:00:59.449733 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:00:59.449752 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:00:59.470729 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:01:06.412761 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:01:06.412784 3798194 memory.go:184] no items to output this cycle I1121 13:01:06.412937 3798194 cpu.go:275] no items to output this cycle E1121 13:01:16.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:01:16.413249 3798194 memory.go:184] no items to output this cycle I1121 13:01:16.413394 3798194 cpu.go:275] no items to output this cycle E1121 13:01:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:01:26.411954 3798194 memory.go:191] Add success. I1121 13:01:26.412314 3798194 cpu.go:282] Add success. I1121 13:01:26.441909 3798194 net.go:642] Add success. I1121 13:01:26.446525 3798194 net.go:778] primary dev: ETH0 I1121 13:01:26.446541 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:01:26.446555 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:01:29.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:01:29.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:01:29.450149 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:01:36.412778 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:01:36.412801 3798194 memory.go:184] no items to output this cycle I1121 13:01:36.412951 3798194 cpu.go:275] no items to output this cycle E1121 13:01:46.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:01:46.411987 3798194 memory.go:184] no items to output this cycle I1121 13:01:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 13:01:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:01:56.411955 3798194 memory.go:191] Add success. W1121 13:01:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:01:56.412003 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:01:56.412006 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:01:56.412341 3798194 cpu.go:282] Add success. I1121 13:01:56.441789 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:01:56.442298 3798194 net.go:642] Add success. I1121 13:01:56.446994 3798194 net.go:778] primary dev: ETH0 I1121 13:01:56.447017 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:01:56.447037 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:01:57.232144 3798194 disk_info.go:125] begin check local disk info of client I1121 13:01:57.235717 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:01:57.235726 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000371a80 0xc000371ac0] I1121 13:01:57.450177 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:01:57.450392 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:01:57.450405 3798194 disk_worker.go:723] disk space is not compliant W1121 13:01:57.450408 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:01:57.458929 3798194 disk_worker.go:509] system disk:vda1 I1121 13:01:57.458971 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:01:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:01:59.450031 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:01:59.450132 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:01:59.450161 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:01:59.471688 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:02:06.412771 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:02:06.412794 3798194 memory.go:184] no items to output this cycle I1121 13:02:06.412946 3798194 cpu.go:275] no items to output this cycle E1121 13:02:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:02:16.413255 3798194 memory.go:184] no items to output this cycle I1121 13:02:16.413399 3798194 cpu.go:275] no items to output this cycle E1121 13:02:26.413252 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:02:26.413284 3798194 memory.go:191] Add success. I1121 13:02:26.417567 3798194 cpu.go:282] Add success. I1121 13:02:26.442077 3798194 net.go:642] Add success. I1121 13:02:26.444231 3798194 net.go:778] primary dev: ETH0 I1121 13:02:26.444247 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:02:26.444261 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:02:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:02:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:02:29.450502 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:02:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:02:36.411975 3798194 memory.go:184] no items to output this cycle I1121 13:02:36.412126 3798194 cpu.go:275] no items to output this cycle E1121 13:02:46.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:02:46.412257 3798194 memory.go:184] no items to output this cycle I1121 13:02:46.412406 3798194 cpu.go:275] no items to output this cycle E1121 13:02:56.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:02:56.412270 3798194 memory.go:191] Add success. W1121 13:02:56.412301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:02:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:02:56.412316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:02:56.416577 3798194 cpu.go:282] Add success. I1121 13:02:56.440900 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:02:56.441401 3798194 net.go:642] Add success. I1121 13:02:56.442982 3798194 net.go:778] primary dev: ETH0 I1121 13:02:56.442997 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:02:56.443009 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:02:57.235873 3798194 disk_info.go:125] begin check local disk info of client I1121 13:02:57.239483 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:02:57.239492 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003932c0 0xc000393300] I1121 13:02:57.445243 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:02:57.445456 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:02:57.445523 3798194 disk_worker.go:723] disk space is not compliant W1121 13:02:57.445526 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:02:57.448039 3798194 disk_worker.go:509] system disk:vda1 I1121 13:02:57.448079 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:02:58.445989 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:02:59.450505 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:02:59.450577 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:02:59.450607 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:02:59.471092 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:03:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:03:06.411986 3798194 memory.go:184] no items to output this cycle I1121 13:03:06.412143 3798194 cpu.go:275] no items to output this cycle E1121 13:03:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:03:16.411980 3798194 memory.go:184] no items to output this cycle I1121 13:03:16.412141 3798194 cpu.go:275] no items to output this cycle I1121 13:03:17.469566 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:03:17.469575 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:03:26.411548 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:03:26.411581 3798194 memory.go:191] Add success. I1121 13:03:26.411896 3798194 cpu.go:282] Add success. I1121 13:03:26.441444 3798194 net.go:642] Add success. I1121 13:03:26.445837 3798194 net.go:778] primary dev: ETH0 I1121 13:03:26.445853 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:03:26.445866 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:03:29.450608 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:03:29.450685 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:03:29.450715 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:03:36.413256 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:03:36.413277 3798194 memory.go:184] no items to output this cycle I1121 13:03:36.413422 3798194 cpu.go:275] no items to output this cycle E1121 13:03:46.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:03:46.412267 3798194 memory.go:184] no items to output this cycle I1121 13:03:46.412415 3798194 cpu.go:275] no items to output this cycle E1121 13:03:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:03:56.413266 3798194 memory.go:191] Add success. W1121 13:03:56.413292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:03:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:03:56.413306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:03:56.417540 3798194 cpu.go:282] Add success. I1121 13:03:56.441073 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:03:56.441703 3798194 net.go:642] Add success. I1121 13:03:56.443363 3798194 net.go:778] primary dev: ETH0 I1121 13:03:56.443380 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:03:56.443400 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:03:57.239597 3798194 disk_info.go:125] begin check local disk info of client I1121 13:03:57.243545 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:03:57.243553 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a2f80 0xc0004a2fc0] I1121 13:03:57.450796 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:03:57.451030 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:03:57.451043 3798194 disk_worker.go:723] disk space is not compliant W1121 13:03:57.451046 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:03:57.459348 3798194 disk_worker.go:509] system disk:vda1 I1121 13:03:57.459390 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:03:58.446018 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:03:59.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:03:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:03:59.450144 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:03:59.471672 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:04:06.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:04:06.412252 3798194 memory.go:184] no items to output this cycle I1121 13:04:06.412405 3798194 cpu.go:275] no items to output this cycle E1121 13:04:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:04:16.411980 3798194 memory.go:184] no items to output this cycle I1121 13:04:16.412136 3798194 cpu.go:275] no items to output this cycle E1121 13:04:26.412871 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:04:26.412913 3798194 memory.go:191] Add success. I1121 13:04:26.417292 3798194 cpu.go:282] Add success. I1121 13:04:26.441997 3798194 net.go:642] Add success. I1121 13:04:26.443490 3798194 net.go:778] primary dev: ETH0 I1121 13:04:26.443506 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:04:26.443520 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:04:29.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:04:29.450493 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:04:29.450521 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:04:36.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:04:36.412891 3798194 memory.go:184] no items to output this cycle I1121 13:04:36.413034 3798194 cpu.go:275] no items to output this cycle E1121 13:04:46.413047 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:04:46.413067 3798194 memory.go:184] no items to output this cycle I1121 13:04:46.413231 3798194 cpu.go:275] no items to output this cycle E1121 13:04:56.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:04:56.413262 3798194 memory.go:191] Add success. W1121 13:04:56.413301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:04:56.413313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:04:56.413316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:04:56.417529 3798194 cpu.go:282] Add success. I1121 13:04:56.441920 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:04:56.442480 3798194 net.go:642] Add success. I1121 13:04:56.443967 3798194 net.go:778] primary dev: ETH0 I1121 13:04:56.443985 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:04:56.443997 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:04:57.246140 3798194 disk_info.go:125] begin check local disk info of client I1121 13:04:57.249751 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:04:57.249760 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d6640 0xc0003d6680] I1121 13:04:57.450593 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:04:57.450833 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:04:57.450849 3798194 disk_worker.go:723] disk space is not compliant W1121 13:04:57.450852 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:04:57.458937 3798194 disk_worker.go:509] system disk:vda1 I1121 13:04:57.458981 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:04:58.446009 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:04:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:04:59.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:04:59.450501 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:04:59.471017 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:05:06.413257 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:05:06.413280 3798194 memory.go:184] no items to output this cycle I1121 13:05:06.413422 3798194 cpu.go:275] no items to output this cycle E1121 13:05:16.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:05:16.413366 3798194 memory.go:184] no items to output this cycle I1121 13:05:16.413696 3798194 cpu.go:275] no items to output this cycle E1121 13:05:26.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:05:26.411947 3798194 memory.go:191] Add success. I1121 13:05:26.412319 3798194 cpu.go:282] Add success. I1121 13:05:26.441950 3798194 net.go:642] Add success. I1121 13:05:26.446163 3798194 net.go:778] primary dev: ETH0 I1121 13:05:26.446178 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:05:26.446192 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:05:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:05:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:05:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:05:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:05:36.413258 3798194 memory.go:184] no items to output this cycle I1121 13:05:36.413394 3798194 cpu.go:275] no items to output this cycle E1121 13:05:46.412779 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:05:46.412807 3798194 memory.go:184] no items to output this cycle I1121 13:05:46.412959 3798194 cpu.go:275] no items to output this cycle E1121 13:05:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:05:56.413267 3798194 memory.go:191] Add success. W1121 13:05:56.413295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:05:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:05:56.413309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 W1121 13:05:56.417519 3798194 conf_downlod.go:84] cant't download conf: Get "": unsupported protocol scheme "" W1121 13:05:56.417531 3798194 conf_downlod.go:89] use old conf I1121 13:05:56.417713 3798194 cpu.go:282] Add success. I1121 13:05:56.443412 3798194 event_worker.go:152] Polling the log file for events... I1121 13:05:56.443725 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:05:56.444207 3798194 net.go:642] Add success. I1121 13:05:56.447005 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795235756, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 13:05:56.447135 3798194 net.go:778] primary dev: ETH0 I1121 13:05:56.447149 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:05:56.447161 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:05:56.458368 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 I1121 13:05:57.249865 3798194 disk_info.go:125] begin check local disk info of client I1121 13:05:57.253507 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:05:57.253516 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000310b80 0xc000310bc0] E1121 13:05:57.445171 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 13:05:57.445188 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:05:57.445497 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:05:57.445511 3798194 disk_worker.go:723] disk space is not compliant W1121 13:05:57.445514 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:05:57.447962 3798194 disk_worker.go:509] system disk:vda1 I1121 13:05:57.448000 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 13:05:58.448173 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 13:05:58.448188 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 13:05:59.448980 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 13:05:59.450056 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:05:59.450117 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:05:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:05:59.471646 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:06:06.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:06:06.412874 3798194 memory.go:184] no items to output this cycle I1121 13:06:06.413010 3798194 cpu.go:275] no items to output this cycle E1121 13:06:16.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:06:16.413249 3798194 memory.go:184] no items to output this cycle I1121 13:06:16.413392 3798194 cpu.go:275] no items to output this cycle I1121 13:06:17.469729 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:06:17.469739 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:06:26.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:06:26.412260 3798194 memory.go:191] Add success. I1121 13:06:26.416481 3798194 cpu.go:282] Add success. I1121 13:06:26.442029 3798194 net.go:642] Add success. I1121 13:06:26.443473 3798194 net.go:778] primary dev: ETH0 I1121 13:06:26.443488 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:06:26.443500 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:06:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:06:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:06:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:06:36.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:06:36.412252 3798194 memory.go:184] no items to output this cycle I1121 13:06:36.412405 3798194 cpu.go:275] no items to output this cycle E1121 13:06:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:06:46.413267 3798194 memory.go:184] no items to output this cycle I1121 13:06:46.413414 3798194 cpu.go:275] no items to output this cycle E1121 13:06:56.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:06:56.412272 3798194 memory.go:191] Add success. W1121 13:06:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:06:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:06:56.412312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:06:56.416601 3798194 cpu.go:282] Add success. I1121 13:06:56.440922 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:06:56.441433 3798194 net.go:642] Add success. I1121 13:06:56.442932 3798194 net.go:778] primary dev: ETH0 I1121 13:06:56.442948 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:06:56.442962 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:06:57.257145 3798194 disk_info.go:125] begin check local disk info of client I1121 13:06:57.260639 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:06:57.260648 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00025c680 0xc00025c6c0] I1121 13:06:57.445618 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:06:57.445877 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:06:57.445889 3798194 disk_worker.go:723] disk space is not compliant W1121 13:06:57.445892 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:06:57.448390 3798194 disk_worker.go:509] system disk:vda1 I1121 13:06:57.448430 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:06:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:06:59.450489 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:06:59.450574 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:06:59.450601 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:06:59.471129 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:07:06.413252 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:07:06.413274 3798194 memory.go:184] no items to output this cycle I1121 13:07:06.413420 3798194 cpu.go:275] no items to output this cycle E1121 13:07:16.413345 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:07:16.413366 3798194 memory.go:184] no items to output this cycle I1121 13:07:16.413514 3798194 cpu.go:275] no items to output this cycle E1121 13:07:26.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:07:26.413274 3798194 memory.go:191] Add success. I1121 13:07:26.418401 3798194 cpu.go:282] Add success. I1121 13:07:26.442046 3798194 net.go:642] Add success. I1121 13:07:26.443650 3798194 net.go:778] primary dev: ETH0 I1121 13:07:26.443665 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:07:26.443680 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:07:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:07:29.450491 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:07:29.450518 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:07:36.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:07:36.412263 3798194 memory.go:184] no items to output this cycle I1121 13:07:36.412405 3798194 cpu.go:275] no items to output this cycle E1121 13:07:46.413253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:07:46.413282 3798194 memory.go:184] no items to output this cycle I1121 13:07:46.413456 3798194 cpu.go:275] no items to output this cycle E1121 13:07:56.412285 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:07:56.412317 3798194 memory.go:191] Add success. W1121 13:07:56.412346 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:07:56.412358 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:07:56.412361 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:07:56.416720 3798194 cpu.go:282] Add success. I1121 13:07:56.441100 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:07:56.441616 3798194 net.go:642] Add success. I1121 13:07:56.443364 3798194 net.go:778] primary dev: ETH0 I1121 13:07:56.443378 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:07:56.443390 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:07:57.260746 3798194 disk_info.go:125] begin check local disk info of client I1121 13:07:57.264311 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:07:57.264320 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000473ec0 0xc000473f00] I1121 13:07:57.450333 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:07:57.450556 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:07:57.450568 3798194 disk_worker.go:723] disk space is not compliant W1121 13:07:57.450571 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:07:57.460570 3798194 disk_worker.go:509] system disk:vda1 I1121 13:07:57.460614 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:07:58.446049 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:07:59.450137 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:07:59.450217 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:07:59.450242 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:07:59.470724 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:08:06.413252 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:08:06.413273 3798194 memory.go:184] no items to output this cycle I1121 13:08:06.413432 3798194 cpu.go:275] no items to output this cycle E1121 13:08:16.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:08:16.412252 3798194 memory.go:184] no items to output this cycle I1121 13:08:16.412405 3798194 cpu.go:275] no items to output this cycle E1121 13:08:26.412874 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:08:26.412915 3798194 memory.go:191] Add success. I1121 13:08:26.417634 3798194 cpu.go:282] Add success. I1121 13:08:26.442235 3798194 net.go:642] Add success. I1121 13:08:26.443830 3798194 net.go:778] primary dev: ETH0 I1121 13:08:26.443846 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:08:26.443859 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:08:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:08:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:08:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:08:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:08:36.411979 3798194 memory.go:184] no items to output this cycle I1121 13:08:36.412129 3798194 cpu.go:275] no items to output this cycle E1121 13:08:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:08:46.411981 3798194 memory.go:184] no items to output this cycle I1121 13:08:46.412137 3798194 cpu.go:275] no items to output this cycle E1121 13:08:56.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:08:56.412891 3798194 memory.go:191] Add success. W1121 13:08:56.412921 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:08:56.412934 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:08:56.412938 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:08:56.417556 3798194 cpu.go:282] Add success. I1121 13:08:56.440945 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:08:56.441522 3798194 net.go:642] Add success. I1121 13:08:56.443198 3798194 net.go:778] primary dev: ETH0 I1121 13:08:56.443213 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:08:56.443225 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:08:57.267146 3798194 disk_info.go:125] begin check local disk info of client I1121 13:08:57.270774 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:08:57.270782 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003af140 0xc0003af180] I1121 13:08:57.450713 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:08:57.450915 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:08:57.450938 3798194 disk_worker.go:723] disk space is not compliant W1121 13:08:57.450942 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:08:57.459090 3798194 disk_worker.go:509] system disk:vda1 I1121 13:08:57.459149 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:08:58.446008 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:08:59.450105 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:08:59.450191 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:08:59.450217 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:08:59.471720 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:09:06.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:09:06.413252 3798194 memory.go:184] no items to output this cycle I1121 13:09:06.413403 3798194 cpu.go:275] no items to output this cycle E1121 13:09:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:09:16.411984 3798194 memory.go:184] no items to output this cycle I1121 13:09:16.412142 3798194 cpu.go:275] no items to output this cycle I1121 13:09:17.470203 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:09:17.470212 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:09:26.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:09:26.412274 3798194 memory.go:191] Add success. I1121 13:09:26.418041 3798194 cpu.go:282] Add success. I1121 13:09:26.441297 3798194 net.go:642] Add success. I1121 13:09:26.442771 3798194 net.go:778] primary dev: ETH0 I1121 13:09:26.442786 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:09:26.442801 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:09:29.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:09:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:09:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:09:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:09:36.411978 3798194 memory.go:184] no items to output this cycle I1121 13:09:36.412132 3798194 cpu.go:275] no items to output this cycle E1121 13:09:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:09:46.413253 3798194 memory.go:184] no items to output this cycle I1121 13:09:46.413395 3798194 cpu.go:275] no items to output this cycle E1121 13:09:56.412939 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:09:56.412975 3798194 memory.go:191] Add success. W1121 13:09:56.413006 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:09:56.413018 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:09:56.413021 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:09:56.417310 3798194 cpu.go:282] Add success. I1121 13:09:56.441679 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:09:56.442353 3798194 net.go:642] Add success. I1121 13:09:56.443899 3798194 net.go:778] primary dev: ETH0 I1121 13:09:56.443913 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:09:56.443925 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:09:57.274151 3798194 disk_info.go:125] begin check local disk info of client I1121 13:09:57.277719 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:09:57.277730 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000451540 0xc000451580] I1121 13:09:57.450576 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:09:57.450784 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:09:57.450794 3798194 disk_worker.go:723] disk space is not compliant W1121 13:09:57.450797 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:09:57.458803 3798194 disk_worker.go:509] system disk:vda1 I1121 13:09:57.458845 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:09:58.446384 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:09:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:09:59.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:09:59.450132 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:09:59.471644 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:10:06.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:10:06.411976 3798194 memory.go:184] no items to output this cycle I1121 13:10:06.412127 3798194 cpu.go:275] no items to output this cycle E1121 13:10:16.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:10:16.413270 3798194 memory.go:184] no items to output this cycle I1121 13:10:16.413419 3798194 cpu.go:275] no items to output this cycle E1121 13:10:26.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:10:26.411952 3798194 memory.go:191] Add success. I1121 13:10:26.412280 3798194 cpu.go:282] Add success. I1121 13:10:26.441748 3798194 net.go:642] Add success. I1121 13:10:26.445660 3798194 net.go:778] primary dev: ETH0 I1121 13:10:26.445676 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:10:26.445690 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:10:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:10:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:10:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:10:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:10:36.413259 3798194 memory.go:184] no items to output this cycle I1121 13:10:36.413400 3798194 cpu.go:275] no items to output this cycle E1121 13:10:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:10:46.413253 3798194 memory.go:184] no items to output this cycle I1121 13:10:46.413404 3798194 cpu.go:275] no items to output this cycle E1121 13:10:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:10:56.413262 3798194 memory.go:191] Add success. W1121 13:10:56.413292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:10:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:10:56.413307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:10:56.417714 3798194 cpu.go:282] Add success. I1121 13:10:56.442365 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:10:56.442832 3798194 net.go:642] Add success. I1121 13:10:56.445708 3798194 net.go:778] primary dev: ETH0 I1121 13:10:56.445723 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:10:56.445736 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:10:57.281151 3798194 disk_info.go:125] begin check local disk info of client I1121 13:10:57.284685 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:10:57.284694 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004b1a80 0xc0004b1ac0] E1121 13:10:57.445691 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 13:10:57.445708 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:10:57.445976 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:10:57.445988 3798194 disk_worker.go:723] disk space is not compliant W1121 13:10:57.445991 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:10:57.448472 3798194 disk_worker.go:509] system disk:vda1 I1121 13:10:57.448512 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 13:10:58.448218 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 13:10:58.448232 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 13:10:59.448610 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 13:10:59.449686 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:10:59.449736 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:10:59.449757 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:10:59.471183 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:11:06.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:11:06.412886 3798194 memory.go:184] no items to output this cycle I1121 13:11:06.413031 3798194 cpu.go:275] no items to output this cycle E1121 13:11:16.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:11:16.412256 3798194 memory.go:184] no items to output this cycle I1121 13:11:16.412407 3798194 cpu.go:275] no items to output this cycle E1121 13:11:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:11:26.413264 3798194 memory.go:191] Add success. I1121 13:11:26.417984 3798194 cpu.go:282] Add success. I1121 13:11:26.441545 3798194 net.go:642] Add success. I1121 13:11:26.443028 3798194 net.go:778] primary dev: ETH0 I1121 13:11:26.443043 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:11:26.443056 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:11:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:11:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:11:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:11:36.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:11:36.413262 3798194 memory.go:184] no items to output this cycle I1121 13:11:36.413403 3798194 cpu.go:275] no items to output this cycle E1121 13:11:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:11:46.413262 3798194 memory.go:184] no items to output this cycle I1121 13:11:46.413408 3798194 cpu.go:275] no items to output this cycle E1121 13:11:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:11:56.413269 3798194 memory.go:191] Add success. W1121 13:11:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:11:56.413316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:11:56.413321 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:11:56.417782 3798194 cpu.go:282] Add success. I1121 13:11:56.441194 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:11:56.441784 3798194 net.go:642] Add success. I1121 13:11:56.443421 3798194 net.go:778] primary dev: ETH0 I1121 13:11:56.443436 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:11:56.443448 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:11:57.288136 3798194 disk_info.go:125] begin check local disk info of client I1121 13:11:57.291621 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:11:57.291630 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000411600 0xc000411640] I1121 13:11:57.450291 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:11:57.450465 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:11:57.450517 3798194 disk_worker.go:723] disk space is not compliant W1121 13:11:57.450521 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:11:57.458573 3798194 disk_worker.go:509] system disk:vda1 I1121 13:11:57.458620 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:11:58.446519 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:11:59.450212 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:11:59.450290 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:11:59.450315 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:11:59.470737 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:12:06.411237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:12:06.411259 3798194 memory.go:184] no items to output this cycle I1121 13:12:06.411499 3798194 cpu.go:275] no items to output this cycle E1121 13:12:16.412255 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:12:16.412277 3798194 memory.go:184] no items to output this cycle I1121 13:12:16.412430 3798194 cpu.go:275] no items to output this cycle I1121 13:12:17.470369 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:12:17.470378 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:12:26.412021 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:12:26.412057 3798194 memory.go:191] Add success. I1121 13:12:26.416238 3798194 cpu.go:282] Add success. I1121 13:12:26.441748 3798194 net.go:642] Add success. I1121 13:12:26.443432 3798194 net.go:778] primary dev: ETH0 I1121 13:12:26.443447 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:12:26.443460 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:12:29.450029 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:12:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:12:29.450154 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:12:36.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:12:36.413257 3798194 memory.go:184] no items to output this cycle I1121 13:12:36.413398 3798194 cpu.go:275] no items to output this cycle E1121 13:12:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:12:46.411975 3798194 memory.go:184] no items to output this cycle I1121 13:12:46.412140 3798194 cpu.go:275] no items to output this cycle E1121 13:12:56.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:12:56.412888 3798194 memory.go:191] Add success. W1121 13:12:56.412920 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:12:56.412932 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:12:56.412935 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:12:56.417409 3798194 cpu.go:282] Add success. I1121 13:12:56.441844 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:12:56.442447 3798194 net.go:642] Add success. I1121 13:12:56.443921 3798194 net.go:778] primary dev: ETH0 I1121 13:12:56.443937 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:12:56.443952 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:12:57.295147 3798194 disk_info.go:125] begin check local disk info of client I1121 13:12:57.298721 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:12:57.298730 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00035ac00 0xc00035ac40] I1121 13:12:57.450525 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:12:57.450701 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:12:57.450759 3798194 disk_worker.go:723] disk space is not compliant W1121 13:12:57.450762 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:12:57.458799 3798194 disk_worker.go:509] system disk:vda1 I1121 13:12:57.458842 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:12:58.446389 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:12:59.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:12:59.450467 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:12:59.450493 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:12:59.471010 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:13:06.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:13:06.413267 3798194 memory.go:184] no items to output this cycle I1121 13:13:06.413413 3798194 cpu.go:275] no items to output this cycle E1121 13:13:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:13:16.411986 3798194 memory.go:184] no items to output this cycle I1121 13:13:16.412144 3798194 cpu.go:275] no items to output this cycle E1121 13:13:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:13:26.411958 3798194 memory.go:191] Add success. I1121 13:13:26.412300 3798194 cpu.go:282] Add success. I1121 13:13:26.441917 3798194 net.go:642] Add success. I1121 13:13:26.446499 3798194 net.go:778] primary dev: ETH0 I1121 13:13:26.446515 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:13:26.446529 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:13:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:13:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:13:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:13:36.412749 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:13:36.412773 3798194 memory.go:184] no items to output this cycle I1121 13:13:36.412921 3798194 cpu.go:275] no items to output this cycle E1121 13:13:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:13:46.411992 3798194 memory.go:184] no items to output this cycle I1121 13:13:46.412149 3798194 cpu.go:275] no items to output this cycle E1121 13:13:56.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:13:56.413271 3798194 memory.go:191] Add success. W1121 13:13:56.413301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:13:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:13:56.413317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:13:56.417941 3798194 cpu.go:282] Add success. I1121 13:13:56.441437 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:13:56.441917 3798194 net.go:642] Add success. I1121 13:13:56.443653 3798194 net.go:778] primary dev: ETH0 I1121 13:13:56.443667 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:13:56.443680 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:13:57.302148 3798194 disk_info.go:125] begin check local disk info of client I1121 13:13:57.305740 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:13:57.305749 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000398d40 0xc000398d80] I1121 13:13:57.445554 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:13:57.445816 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:13:57.445828 3798194 disk_worker.go:723] disk space is not compliant W1121 13:13:57.445831 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:13:57.448236 3798194 disk_worker.go:509] system disk:vda1 I1121 13:13:57.448277 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:13:58.446386 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:13:59.450487 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:13:59.450566 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:13:59.450590 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:13:59.471083 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:14:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:14:06.413256 3798194 memory.go:184] no items to output this cycle I1121 13:14:06.413403 3798194 cpu.go:275] no items to output this cycle E1121 13:14:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:14:16.411983 3798194 memory.go:184] no items to output this cycle I1121 13:14:16.412139 3798194 cpu.go:275] no items to output this cycle E1121 13:14:26.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:14:26.412891 3798194 memory.go:191] Add success. I1121 13:14:26.417817 3798194 cpu.go:282] Add success. I1121 13:14:26.441293 3798194 net.go:642] Add success. I1121 13:14:26.443003 3798194 net.go:778] primary dev: ETH0 I1121 13:14:26.443019 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:14:26.443033 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:14:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:14:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:14:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:14:36.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:14:36.412262 3798194 memory.go:184] no items to output this cycle I1121 13:14:36.412421 3798194 cpu.go:275] no items to output this cycle E1121 13:14:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:14:46.412259 3798194 memory.go:184] no items to output this cycle I1121 13:14:46.412406 3798194 cpu.go:275] no items to output this cycle E1121 13:14:56.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:14:56.413269 3798194 memory.go:191] Add success. W1121 13:14:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:14:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:14:56.413312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:14:56.417527 3798194 cpu.go:282] Add success. I1121 13:14:56.441897 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:14:56.442427 3798194 net.go:642] Add success. I1121 13:14:56.443845 3798194 net.go:778] primary dev: ETH0 I1121 13:14:56.443859 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:14:56.443874 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:14:57.305845 3798194 disk_info.go:125] begin check local disk info of client I1121 13:14:57.309429 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:14:57.309437 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000469c00 0xc000469c40] I1121 13:14:57.450255 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:14:57.450452 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:14:57.450463 3798194 disk_worker.go:723] disk space is not compliant W1121 13:14:57.450466 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:14:57.459203 3798194 disk_worker.go:509] system disk:vda1 I1121 13:14:57.459240 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:14:58.446381 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:14:59.450493 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:14:59.450581 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:14:59.450605 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:14:59.471127 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:15:06.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:15:06.413257 3798194 memory.go:184] no items to output this cycle I1121 13:15:06.413404 3798194 cpu.go:275] no items to output this cycle E1121 13:15:16.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:15:16.412254 3798194 memory.go:184] no items to output this cycle I1121 13:15:16.412407 3798194 cpu.go:275] no items to output this cycle I1121 13:15:17.471263 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:15:17.471272 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:15:26.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:15:26.412264 3798194 memory.go:191] Add success. I1121 13:15:26.420655 3798194 cpu.go:282] Add success. I1121 13:15:26.442193 3798194 net.go:642] Add success. I1121 13:15:26.443764 3798194 net.go:778] primary dev: ETH0 I1121 13:15:26.443780 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:15:26.443803 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:15:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:15:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:15:29.450122 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:15:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:15:36.411983 3798194 memory.go:184] no items to output this cycle I1121 13:15:36.412134 3798194 cpu.go:275] no items to output this cycle E1121 13:15:46.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:15:46.412249 3798194 memory.go:184] no items to output this cycle I1121 13:15:46.412398 3798194 cpu.go:275] no items to output this cycle E1121 13:15:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:15:56.411951 3798194 memory.go:191] Add success. W1121 13:15:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:15:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:15:56.411991 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:15:56.412322 3798194 cpu.go:282] Add success. I1121 13:15:56.441591 3798194 event_worker.go:152] Polling the log file for events... I1121 13:15:56.444438 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:15:56.444919 3798194 net.go:642] Add success. I1121 13:15:56.449045 3798194 net.go:778] primary dev: ETH0 I1121 13:15:56.449060 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:15:56.449072 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:15:57.312145 3798194 disk_info.go:125] begin check local disk info of client I1121 13:15:57.316038 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:15:57.316048 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000398900 0xc000398940] E1121 13:15:57.450425 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 13:15:57.450439 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:15:57.450587 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:15:57.450636 3798194 disk_worker.go:723] disk space is not compliant W1121 13:15:57.450640 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:15:57.458925 3798194 disk_worker.go:509] system disk:vda1 I1121 13:15:57.458980 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 13:15:58.448169 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 13:15:58.448184 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 13:15:59.448603 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 13:15:59.449678 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:15:59.449729 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:15:59.449748 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:15:59.471269 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:16:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:16:06.413255 3798194 memory.go:184] no items to output this cycle I1121 13:16:06.413399 3798194 cpu.go:275] no items to output this cycle E1121 13:16:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:16:16.411981 3798194 memory.go:184] no items to output this cycle I1121 13:16:16.412137 3798194 cpu.go:275] no items to output this cycle E1121 13:16:26.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:16:26.413273 3798194 memory.go:191] Add success. I1121 13:16:26.417628 3798194 cpu.go:282] Add success. I1121 13:16:26.442520 3798194 net.go:642] Add success. I1121 13:16:26.444061 3798194 net.go:778] primary dev: ETH0 I1121 13:16:26.444077 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:16:26.444096 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:16:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:16:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:16:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:16:36.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:16:36.412253 3798194 memory.go:184] no items to output this cycle I1121 13:16:36.412396 3798194 cpu.go:275] no items to output this cycle E1121 13:16:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:16:46.413264 3798194 memory.go:184] no items to output this cycle I1121 13:16:46.413405 3798194 cpu.go:275] no items to output this cycle E1121 13:16:56.413158 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:16:56.413192 3798194 memory.go:191] Add success. W1121 13:16:56.413219 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:16:56.413230 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:16:56.413234 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:16:56.417223 3798194 cpu.go:282] Add success. I1121 13:16:56.441675 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:16:56.442308 3798194 net.go:642] Add success. I1121 13:16:56.443863 3798194 net.go:778] primary dev: ETH0 I1121 13:16:56.443877 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:16:56.443890 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:16:57.319145 3798194 disk_info.go:125] begin check local disk info of client I1121 13:16:57.323017 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:16:57.323034 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003dadc0 0xc0003dae00] I1121 13:16:57.450934 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:16:57.451091 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:16:57.451178 3798194 disk_worker.go:723] disk space is not compliant W1121 13:16:57.451181 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:16:57.459998 3798194 disk_worker.go:509] system disk:vda1 I1121 13:16:57.460042 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:16:58.446679 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:16:59.449678 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:16:59.449755 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:16:59.449781 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:16:59.471313 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:17:06.413256 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:17:06.413278 3798194 memory.go:184] no items to output this cycle I1121 13:17:06.413428 3798194 cpu.go:275] no items to output this cycle E1121 13:17:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:17:16.412268 3798194 memory.go:184] no items to output this cycle I1121 13:17:16.412413 3798194 cpu.go:275] no items to output this cycle E1121 13:17:26.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:17:26.413269 3798194 memory.go:191] Add success. I1121 13:17:26.417932 3798194 cpu.go:282] Add success. I1121 13:17:26.441483 3798194 net.go:642] Add success. I1121 13:17:26.443235 3798194 net.go:778] primary dev: ETH0 I1121 13:17:26.443249 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:17:26.443262 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:17:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:17:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:17:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:17:36.412778 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:17:36.412799 3798194 memory.go:184] no items to output this cycle I1121 13:17:36.412964 3798194 cpu.go:275] no items to output this cycle E1121 13:17:46.413000 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:17:46.413021 3798194 memory.go:184] no items to output this cycle I1121 13:17:46.413186 3798194 cpu.go:275] no items to output this cycle E1121 13:17:56.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:17:56.413288 3798194 memory.go:191] Add success. W1121 13:17:56.413319 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:17:56.413331 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:17:56.413334 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:17:56.417695 3798194 cpu.go:282] Add success. I1121 13:17:56.441082 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:17:56.442270 3798194 net.go:642] Add success. I1121 13:17:56.443966 3798194 net.go:778] primary dev: ETH0 I1121 13:17:56.443986 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:17:56.443999 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:17:57.323121 3798194 disk_info.go:125] begin check local disk info of client I1121 13:17:57.326695 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:17:57.326704 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004f4900 0xc0004f4940] I1121 13:17:57.450432 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:17:57.450589 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:17:57.450643 3798194 disk_worker.go:723] disk space is not compliant W1121 13:17:57.450646 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:17:57.460097 3798194 disk_worker.go:509] system disk:vda1 I1121 13:17:57.460156 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:17:58.446034 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:17:59.450132 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:17:59.450206 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:17:59.450231 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:17:59.470694 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:18:06.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:18:06.413258 3798194 memory.go:184] no items to output this cycle I1121 13:18:06.413421 3798194 cpu.go:275] no items to output this cycle E1121 13:18:16.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:18:16.413268 3798194 memory.go:184] no items to output this cycle I1121 13:18:16.413431 3798194 cpu.go:275] no items to output this cycle I1121 13:18:17.472354 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:18:17.472362 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:18:26.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:18:26.413285 3798194 memory.go:191] Add success. I1121 13:18:26.417542 3798194 cpu.go:282] Add success. I1121 13:18:26.442102 3798194 net.go:642] Add success. I1121 13:18:26.443633 3798194 net.go:778] primary dev: ETH0 I1121 13:18:26.443647 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:18:26.443660 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:18:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:18:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:18:29.450501 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:18:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:18:36.412256 3798194 memory.go:184] no items to output this cycle I1121 13:18:36.412398 3798194 cpu.go:275] no items to output this cycle E1121 13:18:46.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:18:46.412268 3798194 memory.go:184] no items to output this cycle I1121 13:18:46.412413 3798194 cpu.go:275] no items to output this cycle E1121 13:18:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:18:56.412264 3798194 memory.go:191] Add success. W1121 13:18:56.412293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:18:56.412304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:18:56.412307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:18:56.416518 3798194 cpu.go:282] Add success. I1121 13:18:56.441877 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:18:56.442372 3798194 net.go:642] Add success. I1121 13:18:56.444017 3798194 net.go:778] primary dev: ETH0 I1121 13:18:56.444032 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:18:56.444043 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:18:57.331098 3798194 disk_info.go:125] begin check local disk info of client I1121 13:18:57.334590 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:18:57.334599 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000334fc0 0xc000335000] I1121 13:18:57.450225 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:18:57.450390 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:18:57.450445 3798194 disk_worker.go:723] disk space is not compliant W1121 13:18:57.450448 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:18:57.458604 3798194 disk_worker.go:509] system disk:vda1 I1121 13:18:57.458647 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:18:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:18:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:18:59.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:18:59.450495 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:18:59.470999 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:19:06.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:19:06.413255 3798194 memory.go:184] no items to output this cycle I1121 13:19:06.413589 3798194 cpu.go:275] no items to output this cycle E1121 13:19:16.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:19:16.413250 3798194 memory.go:184] no items to output this cycle I1121 13:19:16.413405 3798194 cpu.go:275] no items to output this cycle E1121 13:19:26.412877 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:19:26.412916 3798194 memory.go:191] Add success. I1121 13:19:26.416983 3798194 cpu.go:282] Add success. I1121 13:19:26.441517 3798194 net.go:642] Add success. I1121 13:19:26.443003 3798194 net.go:778] primary dev: ETH0 I1121 13:19:26.443020 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:19:26.443034 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:19:29.449859 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:19:29.449940 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:19:29.449969 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:19:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:19:36.412259 3798194 memory.go:184] no items to output this cycle I1121 13:19:36.412402 3798194 cpu.go:275] no items to output this cycle E1121 13:19:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:19:46.412258 3798194 memory.go:184] no items to output this cycle I1121 13:19:46.412398 3798194 cpu.go:275] no items to output this cycle E1121 13:19:56.412865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:19:56.412898 3798194 memory.go:191] Add success. W1121 13:19:56.412928 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:19:56.412942 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:19:56.412945 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:19:56.417402 3798194 cpu.go:282] Add success. I1121 13:19:56.441776 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:19:56.442282 3798194 net.go:642] Add success. I1121 13:19:56.443745 3798194 net.go:778] primary dev: ETH0 I1121 13:19:56.443760 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:19:56.443773 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:19:57.338147 3798194 disk_info.go:125] begin check local disk info of client I1121 13:19:57.342099 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:19:57.342108 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00034f640 0xc00034f680] I1121 13:19:57.450077 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:19:57.450302 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:19:57.450313 3798194 disk_worker.go:723] disk space is not compliant W1121 13:19:57.450316 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:19:57.458717 3798194 disk_worker.go:509] system disk:vda1 I1121 13:19:57.458760 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:19:58.446381 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:19:59.450486 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:19:59.450564 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:19:59.450593 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:19:59.471107 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:20:06.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:20:06.413263 3798194 memory.go:184] no items to output this cycle I1121 13:20:06.413404 3798194 cpu.go:275] no items to output this cycle E1121 13:20:16.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:20:16.413255 3798194 memory.go:184] no items to output this cycle I1121 13:20:16.413398 3798194 cpu.go:275] no items to output this cycle E1121 13:20:26.417798 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:20:26.417833 3798194 memory.go:191] Add success. I1121 13:20:26.422158 3798194 cpu.go:282] Add success. I1121 13:20:26.441660 3798194 net.go:642] Add success. I1121 13:20:26.443485 3798194 net.go:778] primary dev: ETH0 I1121 13:20:26.443500 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:20:26.443514 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:20:29.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:20:29.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:20:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:20:36.412356 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:20:36.412382 3798194 memory.go:184] no items to output this cycle I1121 13:20:36.412658 3798194 cpu.go:275] no items to output this cycle E1121 13:20:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:20:46.412258 3798194 memory.go:184] no items to output this cycle I1121 13:20:46.412406 3798194 cpu.go:275] no items to output this cycle E1121 13:20:56.412601 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:20:56.412635 3798194 memory.go:191] Add success. W1121 13:20:56.412664 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:20:56.412675 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:20:56.412679 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:20:56.420149 3798194 cpu.go:282] Add success. I1121 13:20:56.443991 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:20:56.444499 3798194 net.go:642] Add success. I1121 13:20:56.445967 3798194 net.go:778] primary dev: ETH0 I1121 13:20:56.445982 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:20:56.445994 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:20:57.342232 3798194 disk_info.go:125] begin check local disk info of client I1121 13:20:57.345836 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:20:57.345844 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00034ea80 0xc00034eac0] E1121 13:20:57.450502 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 13:20:57.450515 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:20:57.450714 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:20:57.450722 3798194 disk_worker.go:723] disk space is not compliant W1121 13:20:57.450725 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:20:57.458775 3798194 disk_worker.go:509] system disk:vda1 I1121 13:20:57.458818 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 13:20:58.448159 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 13:20:58.448173 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 13:20:59.448985 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 13:20:59.450059 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:20:59.450121 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:20:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:20:59.471678 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:21:06.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:21:06.413259 3798194 memory.go:184] no items to output this cycle I1121 13:21:06.413398 3798194 cpu.go:275] no items to output this cycle E1121 13:21:16.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:21:16.412249 3798194 memory.go:184] no items to output this cycle I1121 13:21:16.412406 3798194 cpu.go:275] no items to output this cycle I1121 13:21:17.472514 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:21:17.472522 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:21:26.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:21:26.412263 3798194 memory.go:191] Add success. I1121 13:21:26.416487 3798194 cpu.go:282] Add success. I1121 13:21:26.442160 3798194 net.go:642] Add success. I1121 13:21:26.443678 3798194 net.go:778] primary dev: ETH0 I1121 13:21:26.443692 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:21:26.443717 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:21:29.449667 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:21:29.449744 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:21:29.449772 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:21:36.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:21:36.412259 3798194 memory.go:184] no items to output this cycle I1121 13:21:36.412404 3798194 cpu.go:275] no items to output this cycle E1121 13:21:46.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:21:46.413261 3798194 memory.go:184] no items to output this cycle I1121 13:21:46.413405 3798194 cpu.go:275] no items to output this cycle E1121 13:21:56.411935 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:21:56.411967 3798194 memory.go:191] Add success. W1121 13:21:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:21:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:21:56.412007 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:21:56.412341 3798194 cpu.go:282] Add success. I1121 13:21:56.441801 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:21:56.442551 3798194 net.go:642] Add success. I1121 13:21:56.446591 3798194 net.go:778] primary dev: ETH0 I1121 13:21:56.446609 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:21:56.446623 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:21:57.345953 3798194 disk_info.go:125] begin check local disk info of client I1121 13:21:57.349486 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:21:57.349495 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046d080 0xc00046d0c0] I1121 13:21:57.450375 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:21:57.450581 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:21:57.450592 3798194 disk_worker.go:723] disk space is not compliant W1121 13:21:57.450595 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:21:57.458691 3798194 disk_worker.go:509] system disk:vda1 I1121 13:21:57.458734 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:21:58.446385 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:21:59.450032 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:21:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:21:59.450146 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:21:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:22:06.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:22:06.413264 3798194 memory.go:184] no items to output this cycle I1121 13:22:06.413421 3798194 cpu.go:275] no items to output this cycle E1121 13:22:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:22:16.411977 3798194 memory.go:184] no items to output this cycle I1121 13:22:16.412134 3798194 cpu.go:275] no items to output this cycle E1121 13:22:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:22:26.413267 3798194 memory.go:191] Add success. I1121 13:22:26.417465 3798194 cpu.go:282] Add success. I1121 13:22:26.441981 3798194 net.go:642] Add success. I1121 13:22:26.443600 3798194 net.go:778] primary dev: ETH0 I1121 13:22:26.443615 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:22:26.443628 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:22:29.450409 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:22:29.450491 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:22:29.450517 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:22:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:22:36.412259 3798194 memory.go:184] no items to output this cycle I1121 13:22:36.412403 3798194 cpu.go:275] no items to output this cycle E1121 13:22:46.411865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:22:46.411886 3798194 memory.go:184] no items to output this cycle I1121 13:22:46.412043 3798194 cpu.go:275] no items to output this cycle E1121 13:22:56.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:22:56.411963 3798194 memory.go:191] Add success. W1121 13:22:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:22:56.412001 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:22:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:22:56.412334 3798194 cpu.go:282] Add success. I1121 13:22:56.441807 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:22:56.442336 3798194 net.go:642] Add success. I1121 13:22:56.454262 3798194 net.go:778] primary dev: ETH0 I1121 13:22:56.454281 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:22:56.454293 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:22:57.353138 3798194 disk_info.go:125] begin check local disk info of client I1121 13:22:57.356761 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:22:57.356771 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d4680 0xc0003d46c0] I1121 13:22:57.450417 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:22:57.450584 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:22:57.450635 3798194 disk_worker.go:723] disk space is not compliant W1121 13:22:57.450639 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:22:57.458992 3798194 disk_worker.go:509] system disk:vda1 I1121 13:22:57.459036 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:22:58.445992 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:22:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:22:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:22:59.450128 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:22:59.471653 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:23:06.412977 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:23:06.412999 3798194 memory.go:184] no items to output this cycle I1121 13:23:06.413173 3798194 cpu.go:275] no items to output this cycle E1121 13:23:16.412360 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:23:16.412385 3798194 memory.go:184] no items to output this cycle I1121 13:23:16.412664 3798194 cpu.go:275] no items to output this cycle E1121 13:23:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:23:26.411956 3798194 memory.go:191] Add success. I1121 13:23:26.412314 3798194 cpu.go:282] Add success. I1121 13:23:26.441966 3798194 net.go:642] Add success. I1121 13:23:26.445824 3798194 net.go:778] primary dev: ETH0 I1121 13:23:26.445839 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:23:26.445854 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:23:29.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:23:29.450489 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:23:29.450517 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:23:36.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:23:36.413248 3798194 memory.go:184] no items to output this cycle I1121 13:23:36.413390 3798194 cpu.go:275] no items to output this cycle E1121 13:23:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:23:46.411985 3798194 memory.go:184] no items to output this cycle I1121 13:23:46.412144 3798194 cpu.go:275] no items to output this cycle E1121 13:23:56.412249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:23:56.412282 3798194 memory.go:191] Add success. W1121 13:23:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:23:56.412322 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:23:56.412325 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:23:56.416326 3798194 cpu.go:282] Add success. I1121 13:23:56.441703 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:23:56.442198 3798194 net.go:642] Add success. I1121 13:23:56.443702 3798194 net.go:778] primary dev: ETH0 I1121 13:23:56.443718 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:23:56.443730 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:23:57.360148 3798194 disk_info.go:125] begin check local disk info of client I1121 13:23:57.363698 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:23:57.363707 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030a700 0xc00030a740] I1121 13:23:57.450403 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:23:57.450564 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:23:57.450619 3798194 disk_worker.go:723] disk space is not compliant W1121 13:23:57.450622 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:23:57.458664 3798194 disk_worker.go:509] system disk:vda1 I1121 13:23:57.458704 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:23:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:23:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:23:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:23:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:23:59.471638 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:24:06.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:24:06.412266 3798194 memory.go:184] no items to output this cycle I1121 13:24:06.412407 3798194 cpu.go:275] no items to output this cycle E1121 13:24:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:24:16.411975 3798194 memory.go:184] no items to output this cycle I1121 13:24:16.412130 3798194 cpu.go:275] no items to output this cycle I1121 13:24:17.473203 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:24:17.473211 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:24:26.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:24:26.412263 3798194 memory.go:191] Add success. I1121 13:24:26.416504 3798194 cpu.go:282] Add success. I1121 13:24:26.442019 3798194 net.go:642] Add success. I1121 13:24:26.443619 3798194 net.go:778] primary dev: ETH0 I1121 13:24:26.443634 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:24:26.443648 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:24:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:24:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:24:29.450493 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:24:36.411233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:24:36.411257 3798194 memory.go:184] no items to output this cycle I1121 13:24:36.411403 3798194 cpu.go:275] no items to output this cycle E1121 13:24:46.412003 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:24:46.412036 3798194 memory.go:184] no items to output this cycle I1121 13:24:46.412250 3798194 cpu.go:275] no items to output this cycle E1121 13:24:56.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:24:56.413274 3798194 memory.go:191] Add success. W1121 13:24:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:24:56.413318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:24:56.413321 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:24:56.417177 3798194 cpu.go:282] Add success. I1121 13:24:56.441441 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:24:56.442026 3798194 net.go:642] Add success. I1121 13:24:56.443508 3798194 net.go:778] primary dev: ETH0 I1121 13:24:56.443524 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:24:56.443536 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:24:57.363810 3798194 disk_info.go:125] begin check local disk info of client I1121 13:24:57.367434 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:24:57.367443 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036e840 0xc00036e880] I1121 13:24:57.449986 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:24:57.450176 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:24:57.450234 3798194 disk_worker.go:723] disk space is not compliant W1121 13:24:57.450237 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:24:57.458273 3798194 disk_worker.go:509] system disk:vda1 I1121 13:24:57.458313 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:24:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:24:59.450490 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:24:59.450576 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:24:59.450602 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:24:59.471105 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:25:06.412867 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:25:06.412889 3798194 memory.go:184] no items to output this cycle I1121 13:25:06.413037 3798194 cpu.go:275] no items to output this cycle E1121 13:25:16.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:25:16.413254 3798194 memory.go:184] no items to output this cycle I1121 13:25:16.413400 3798194 cpu.go:275] no items to output this cycle E1121 13:25:26.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:25:26.412265 3798194 memory.go:191] Add success. I1121 13:25:26.416695 3798194 cpu.go:282] Add success. I1121 13:25:26.442249 3798194 net.go:642] Add success. I1121 13:25:26.443765 3798194 net.go:778] primary dev: ETH0 I1121 13:25:26.443780 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:25:26.443794 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:25:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:25:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:25:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:25:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:25:36.411980 3798194 memory.go:184] no items to output this cycle I1121 13:25:36.412134 3798194 cpu.go:275] no items to output this cycle E1121 13:25:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:25:46.413261 3798194 memory.go:184] no items to output this cycle I1121 13:25:46.413411 3798194 cpu.go:275] no items to output this cycle E1121 13:25:56.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:25:56.412888 3798194 memory.go:191] Add success. W1121 13:25:56.412917 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:25:56.412928 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:25:56.412931 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:25:56.417475 3798194 cpu.go:282] Add success. I1121 13:25:56.441665 3798194 event_worker.go:152] Polling the log file for events... I1121 13:25:56.444501 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:25:56.445074 3798194 net.go:642] Add success. I1121 13:25:56.446762 3798194 net.go:778] primary dev: ETH0 I1121 13:25:56.446778 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:25:56.446790 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:25:57.367537 3798194 disk_info.go:125] begin check local disk info of client I1121 13:25:57.371150 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:25:57.371160 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000443e40 0xc000443e80] E1121 13:25:57.451484 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 13:25:57.451498 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:25:57.451665 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:25:57.451715 3798194 disk_worker.go:723] disk space is not compliant W1121 13:25:57.451719 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:25:57.459869 3798194 disk_worker.go:509] system disk:vda1 I1121 13:25:57.459907 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 13:25:58.447764 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 13:25:58.447777 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 13:25:59.448603 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 13:25:59.449679 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:25:59.449728 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:25:59.449748 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:25:59.471185 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:26:06.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:26:06.411970 3798194 memory.go:184] no items to output this cycle I1121 13:26:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 13:26:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:26:16.411978 3798194 memory.go:184] no items to output this cycle I1121 13:26:16.412145 3798194 cpu.go:275] no items to output this cycle E1121 13:26:26.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:26:26.413274 3798194 memory.go:191] Add success. I1121 13:26:26.417666 3798194 cpu.go:282] Add success. I1121 13:26:26.442217 3798194 net.go:642] Add success. I1121 13:26:26.443586 3798194 net.go:778] primary dev: ETH0 I1121 13:26:26.443601 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:26:26.443614 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:26:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:26:29.450118 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:26:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:26:36.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:26:36.413258 3798194 memory.go:184] no items to output this cycle I1121 13:26:36.413407 3798194 cpu.go:275] no items to output this cycle E1121 13:26:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:26:46.412254 3798194 memory.go:184] no items to output this cycle I1121 13:26:46.412403 3798194 cpu.go:275] no items to output this cycle E1121 13:26:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:26:56.411951 3798194 memory.go:191] Add success. W1121 13:26:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:26:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:26:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:26:56.412343 3798194 cpu.go:282] Add success. I1121 13:26:56.441774 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:26:56.442398 3798194 net.go:642] Add success. I1121 13:26:56.447176 3798194 net.go:778] primary dev: ETH0 I1121 13:26:56.447193 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:26:56.447206 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:26:57.371262 3798194 disk_info.go:125] begin check local disk info of client I1121 13:26:57.374850 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:26:57.374859 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000395400 0xc000395440] I1121 13:26:57.450063 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:26:57.450236 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:26:57.450293 3798194 disk_worker.go:723] disk space is not compliant W1121 13:26:57.450296 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:26:57.458243 3798194 disk_worker.go:509] system disk:vda1 I1121 13:26:57.458285 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:26:58.446385 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:26:59.450484 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:26:59.450556 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:26:59.450584 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:26:59.471126 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:27:06.413162 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:27:06.413184 3798194 memory.go:184] no items to output this cycle I1121 13:27:06.413340 3798194 cpu.go:275] no items to output this cycle E1121 13:27:16.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:27:16.413365 3798194 memory.go:184] no items to output this cycle I1121 13:27:16.413694 3798194 cpu.go:275] no items to output this cycle I1121 13:27:17.474027 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:27:17.474035 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:27:26.413265 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:27:26.413300 3798194 memory.go:191] Add success. I1121 13:27:26.417559 3798194 cpu.go:282] Add success. I1121 13:27:26.442088 3798194 net.go:642] Add success. I1121 13:27:26.443637 3798194 net.go:778] primary dev: ETH0 I1121 13:27:26.443651 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:27:26.443664 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:27:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:27:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:27:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:27:36.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:27:36.411989 3798194 memory.go:184] no items to output this cycle I1121 13:27:36.412142 3798194 cpu.go:275] no items to output this cycle E1121 13:27:46.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:27:46.412255 3798194 memory.go:184] no items to output this cycle I1121 13:27:46.412400 3798194 cpu.go:275] no items to output this cycle E1121 13:27:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:27:56.411948 3798194 memory.go:191] Add success. W1121 13:27:56.411975 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:27:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:27:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:27:56.412336 3798194 cpu.go:282] Add success. I1121 13:27:56.441790 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:27:56.442374 3798194 net.go:642] Add success. I1121 13:27:56.446496 3798194 net.go:778] primary dev: ETH0 I1121 13:27:56.446513 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:27:56.446532 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:27:57.378840 3798194 disk_info.go:125] begin check local disk info of client I1121 13:27:57.382397 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:27:57.382405 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000322700 0xc000322740] I1121 13:27:57.449745 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:27:57.449954 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:27:57.449965 3798194 disk_worker.go:723] disk space is not compliant W1121 13:27:57.449968 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:27:57.458067 3798194 disk_worker.go:509] system disk:vda1 I1121 13:27:57.458122 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:27:58.446056 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:27:59.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:27:59.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:27:59.450511 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:27:59.471014 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:28:06.413172 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:28:06.413194 3798194 memory.go:184] no items to output this cycle I1121 13:28:06.413337 3798194 cpu.go:275] no items to output this cycle E1121 13:28:16.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:28:16.412253 3798194 memory.go:184] no items to output this cycle I1121 13:28:16.412412 3798194 cpu.go:275] no items to output this cycle E1121 13:28:26.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:28:26.412277 3798194 memory.go:191] Add success. I1121 13:28:26.416971 3798194 cpu.go:282] Add success. I1121 13:28:26.441552 3798194 net.go:642] Add success. I1121 13:28:26.443131 3798194 net.go:778] primary dev: ETH0 I1121 13:28:26.443146 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:28:26.443161 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:28:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:28:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:28:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:28:36.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:28:36.411974 3798194 memory.go:184] no items to output this cycle I1121 13:28:36.412136 3798194 cpu.go:275] no items to output this cycle E1121 13:28:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:28:46.411988 3798194 memory.go:184] no items to output this cycle I1121 13:28:46.412148 3798194 cpu.go:275] no items to output this cycle E1121 13:28:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:28:56.411958 3798194 memory.go:191] Add success. W1121 13:28:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:28:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:28:56.412008 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:28:56.412365 3798194 cpu.go:282] Add success. I1121 13:28:56.441925 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:28:56.442529 3798194 net.go:642] Add success. I1121 13:28:56.446550 3798194 net.go:778] primary dev: ETH0 I1121 13:28:56.446565 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:28:56.446580 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:28:57.386804 3798194 disk_info.go:125] begin check local disk info of client I1121 13:28:57.390439 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:28:57.390448 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000322100 0xc000322140] I1121 13:28:57.449500 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:28:57.449699 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:28:57.449710 3798194 disk_worker.go:723] disk space is not compliant W1121 13:28:57.449713 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:28:57.457914 3798194 disk_worker.go:509] system disk:vda1 I1121 13:28:57.457956 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:28:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:28:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:28:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:28:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:28:59.471652 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:29:06.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:29:06.412261 3798194 memory.go:184] no items to output this cycle I1121 13:29:06.412417 3798194 cpu.go:275] no items to output this cycle E1121 13:29:16.412224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:29:16.412245 3798194 memory.go:184] no items to output this cycle I1121 13:29:16.412395 3798194 cpu.go:275] no items to output this cycle E1121 13:29:26.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:29:26.412278 3798194 memory.go:191] Add success. I1121 13:29:26.416676 3798194 cpu.go:282] Add success. I1121 13:29:26.441297 3798194 net.go:642] Add success. I1121 13:29:26.442875 3798194 net.go:778] primary dev: ETH0 I1121 13:29:26.442889 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:29:26.442904 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:29:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:29:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:29:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:29:36.411824 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:29:36.411847 3798194 memory.go:184] no items to output this cycle I1121 13:29:36.411994 3798194 cpu.go:275] no items to output this cycle E1121 13:29:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:29:46.411984 3798194 memory.go:184] no items to output this cycle I1121 13:29:46.412147 3798194 cpu.go:275] no items to output this cycle E1121 13:29:56.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:29:56.412262 3798194 memory.go:191] Add success. W1121 13:29:56.412289 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:29:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:29:56.412303 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:29:56.416810 3798194 cpu.go:282] Add success. I1121 13:29:56.441199 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:29:56.441745 3798194 net.go:642] Add success. I1121 13:29:56.443478 3798194 net.go:778] primary dev: ETH0 I1121 13:29:56.443492 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:29:56.443505 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:29:57.394851 3798194 disk_info.go:125] begin check local disk info of client I1121 13:29:57.398405 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:29:57.398413 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000441e80 0xc000441ec0] I1121 13:29:57.448518 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:29:57.448745 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:29:57.448756 3798194 disk_worker.go:723] disk space is not compliant W1121 13:29:57.448759 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:29:57.452314 3798194 disk_worker.go:509] system disk:vda1 I1121 13:29:57.452364 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:29:58.446008 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:29:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:29:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:29:59.450125 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:29:59.471663 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:30:06.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:30:06.411989 3798194 memory.go:184] no items to output this cycle I1121 13:30:06.412144 3798194 cpu.go:275] no items to output this cycle E1121 13:30:16.412246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:30:16.412269 3798194 memory.go:184] no items to output this cycle I1121 13:30:16.412412 3798194 cpu.go:275] no items to output this cycle I1121 13:30:17.474808 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:30:17.474816 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:30:26.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:30:26.412271 3798194 memory.go:191] Add success. I1121 13:30:26.416452 3798194 cpu.go:282] Add success. I1121 13:30:26.442003 3798194 net.go:642] Add success. I1121 13:30:26.443640 3798194 net.go:778] primary dev: ETH0 I1121 13:30:26.443656 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:30:26.443669 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:30:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:30:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:30:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:30:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:30:36.411975 3798194 memory.go:184] no items to output this cycle I1121 13:30:36.412135 3798194 cpu.go:275] no items to output this cycle E1121 13:30:46.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:30:46.413264 3798194 memory.go:184] no items to output this cycle I1121 13:30:46.413419 3798194 cpu.go:275] no items to output this cycle E1121 13:30:56.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:30:56.411959 3798194 memory.go:191] Add success. W1121 13:30:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:30:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:30:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:30:56.412347 3798194 cpu.go:282] Add success. I1121 13:30:56.444484 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:30:56.445081 3798194 net.go:642] Add success. I1121 13:30:56.448749 3798194 net.go:778] primary dev: ETH0 I1121 13:30:56.448766 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:30:56.448779 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:30:57.402142 3798194 disk_info.go:125] begin check local disk info of client I1121 13:30:57.405687 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:30:57.405700 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003cd380 0xc0003cd3c0] E1121 13:30:57.450016 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 13:30:57.450029 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:30:57.450194 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:30:57.450250 3798194 disk_worker.go:723] disk space is not compliant W1121 13:30:57.450253 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:30:57.458334 3798194 disk_worker.go:509] system disk:vda1 I1121 13:30:57.458377 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 13:30:58.447852 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 13:30:58.447867 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 13:30:59.448645 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 13:30:59.449722 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:30:59.449775 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:30:59.449795 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:30:59.471336 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:31:06.412783 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:31:06.412806 3798194 memory.go:184] no items to output this cycle I1121 13:31:06.412957 3798194 cpu.go:275] no items to output this cycle E1121 13:31:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:31:16.412077 3798194 memory.go:184] no items to output this cycle I1121 13:31:16.412426 3798194 cpu.go:275] no items to output this cycle E1121 13:31:26.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:31:26.412279 3798194 memory.go:191] Add success. I1121 13:31:26.416362 3798194 cpu.go:282] Add success. I1121 13:31:26.441915 3798194 net.go:642] Add success. I1121 13:31:26.443546 3798194 net.go:778] primary dev: ETH0 I1121 13:31:26.443561 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:31:26.443575 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:31:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:31:29.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:31:29.450159 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:31:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:31:36.411980 3798194 memory.go:184] no items to output this cycle I1121 13:31:36.412132 3798194 cpu.go:275] no items to output this cycle E1121 13:31:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:31:46.412257 3798194 memory.go:184] no items to output this cycle I1121 13:31:46.412408 3798194 cpu.go:275] no items to output this cycle E1121 13:31:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:31:56.411955 3798194 memory.go:191] Add success. W1121 13:31:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:31:56.412001 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:31:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:31:56.412362 3798194 cpu.go:282] Add success. I1121 13:31:56.441799 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:31:56.442334 3798194 net.go:642] Add success. I1121 13:31:56.446356 3798194 net.go:778] primary dev: ETH0 I1121 13:31:56.446372 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:31:56.446384 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:31:57.409832 3798194 disk_info.go:125] begin check local disk info of client I1121 13:31:57.413365 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:31:57.413374 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002fe700 0xc0002fe740] I1121 13:31:57.450755 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:31:57.450959 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:31:57.450969 3798194 disk_worker.go:723] disk space is not compliant W1121 13:31:57.450972 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:31:57.459454 3798194 disk_worker.go:509] system disk:vda1 I1121 13:31:57.459497 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:31:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:31:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:31:59.450124 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:31:59.450153 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:31:59.471672 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:32:06.412091 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:32:06.412130 3798194 memory.go:184] no items to output this cycle I1121 13:32:06.412281 3798194 cpu.go:275] no items to output this cycle E1121 13:32:16.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:32:16.412255 3798194 memory.go:184] no items to output this cycle I1121 13:32:16.412400 3798194 cpu.go:275] no items to output this cycle E1121 13:32:26.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:32:26.413273 3798194 memory.go:191] Add success. I1121 13:32:26.417623 3798194 cpu.go:282] Add success. I1121 13:32:26.441217 3798194 net.go:642] Add success. I1121 13:32:26.442883 3798194 net.go:778] primary dev: ETH0 I1121 13:32:26.442900 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:32:26.442913 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:32:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:32:29.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:32:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:32:36.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:32:36.413264 3798194 memory.go:184] no items to output this cycle I1121 13:32:36.413408 3798194 cpu.go:275] no items to output this cycle E1121 13:32:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:32:46.412264 3798194 memory.go:184] no items to output this cycle I1121 13:32:46.412420 3798194 cpu.go:275] no items to output this cycle E1121 13:32:56.411932 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:32:56.411962 3798194 memory.go:191] Add success. W1121 13:32:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:32:56.412003 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:32:56.412007 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:32:56.412364 3798194 cpu.go:282] Add success. I1121 13:32:56.441792 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:32:56.442295 3798194 net.go:642] Add success. I1121 13:32:56.446464 3798194 net.go:778] primary dev: ETH0 I1121 13:32:56.446480 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:32:56.446494 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:32:57.413869 3798194 disk_info.go:125] begin check local disk info of client I1121 13:32:57.417403 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:32:57.417412 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000344840 0xc000344880] I1121 13:32:57.450218 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:32:57.450375 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:32:57.450429 3798194 disk_worker.go:723] disk space is not compliant W1121 13:32:57.450432 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:32:57.458442 3798194 disk_worker.go:509] system disk:vda1 I1121 13:32:57.458483 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:32:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:32:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:32:59.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:32:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:32:59.471654 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:33:06.412775 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:33:06.412797 3798194 memory.go:184] no items to output this cycle I1121 13:33:06.412947 3798194 cpu.go:275] no items to output this cycle E1121 13:33:16.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:33:16.412254 3798194 memory.go:184] no items to output this cycle I1121 13:33:16.412401 3798194 cpu.go:275] no items to output this cycle I1121 13:33:17.475592 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:33:17.475601 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:33:26.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:33:26.412263 3798194 memory.go:191] Add success. I1121 13:33:26.416279 3798194 cpu.go:282] Add success. I1121 13:33:26.441760 3798194 net.go:642] Add success. I1121 13:33:26.443211 3798194 net.go:778] primary dev: ETH0 I1121 13:33:26.443225 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:33:26.443240 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:33:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:33:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:33:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:33:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:33:36.411980 3798194 memory.go:184] no items to output this cycle I1121 13:33:36.412136 3798194 cpu.go:275] no items to output this cycle E1121 13:33:46.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:33:46.413251 3798194 memory.go:184] no items to output this cycle I1121 13:33:46.413399 3798194 cpu.go:275] no items to output this cycle E1121 13:33:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:33:56.413263 3798194 memory.go:191] Add success. W1121 13:33:56.413292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:33:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:33:56.413307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:33:56.417531 3798194 cpu.go:282] Add success. I1121 13:33:56.441906 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:33:56.442424 3798194 net.go:642] Add success. I1121 13:33:56.443940 3798194 net.go:778] primary dev: ETH0 I1121 13:33:56.443954 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:33:56.443968 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:33:57.417554 3798194 disk_info.go:125] begin check local disk info of client I1121 13:33:57.421169 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:33:57.421177 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00045de00 0xc00045de40] I1121 13:33:57.450004 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:33:57.450228 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:33:57.450239 3798194 disk_worker.go:723] disk space is not compliant W1121 13:33:57.450242 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:33:57.458895 3798194 disk_worker.go:509] system disk:vda1 I1121 13:33:57.458932 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:33:58.446134 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:33:59.450237 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:33:59.450320 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:33:59.450345 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:33:59.470891 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:34:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:34:06.412255 3798194 memory.go:184] no items to output this cycle I1121 13:34:06.412402 3798194 cpu.go:275] no items to output this cycle E1121 13:34:16.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:34:16.412251 3798194 memory.go:184] no items to output this cycle I1121 13:34:16.412400 3798194 cpu.go:275] no items to output this cycle E1121 13:34:26.412323 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:34:26.412356 3798194 memory.go:191] Add success. I1121 13:34:26.416989 3798194 cpu.go:282] Add success. I1121 13:34:26.441527 3798194 net.go:642] Add success. I1121 13:34:26.442993 3798194 net.go:778] primary dev: ETH0 I1121 13:34:26.443008 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:34:26.443022 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:34:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:34:29.450118 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:34:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:34:36.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:34:36.411987 3798194 memory.go:184] no items to output this cycle I1121 13:34:36.412144 3798194 cpu.go:275] no items to output this cycle E1121 13:34:46.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:34:46.412885 3798194 memory.go:184] no items to output this cycle I1121 13:34:46.413038 3798194 cpu.go:275] no items to output this cycle E1121 13:34:56.412251 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:34:56.412291 3798194 memory.go:191] Add success. W1121 13:34:56.412319 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:34:56.412331 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:34:56.412334 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:34:56.416511 3798194 cpu.go:282] Add success. I1121 13:34:56.441884 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:34:56.442386 3798194 net.go:642] Add success. I1121 13:34:56.443853 3798194 net.go:778] primary dev: ETH0 I1121 13:34:56.443867 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:34:56.443879 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:34:57.421888 3798194 disk_info.go:125] begin check local disk info of client I1121 13:34:57.425437 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:34:57.425446 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000386900 0xc000386940] I1121 13:34:57.450265 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:34:57.450480 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:34:57.450490 3798194 disk_worker.go:723] disk space is not compliant W1121 13:34:57.450493 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:34:57.458518 3798194 disk_worker.go:509] system disk:vda1 I1121 13:34:57.458560 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:34:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:34:59.450116 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:34:59.450196 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:34:59.450221 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:34:59.470707 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:35:06.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:35:06.412883 3798194 memory.go:184] no items to output this cycle I1121 13:35:06.413023 3798194 cpu.go:275] no items to output this cycle E1121 13:35:16.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:35:16.413260 3798194 memory.go:184] no items to output this cycle I1121 13:35:16.413406 3798194 cpu.go:275] no items to output this cycle E1121 13:35:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:35:26.411948 3798194 memory.go:191] Add success. I1121 13:35:26.412307 3798194 cpu.go:282] Add success. I1121 13:35:26.442334 3798194 net.go:642] Add success. I1121 13:35:26.446626 3798194 net.go:778] primary dev: ETH0 I1121 13:35:26.446647 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:35:26.446747 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:35:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:35:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:35:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:35:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:35:36.412254 3798194 memory.go:184] no items to output this cycle I1121 13:35:36.412395 3798194 cpu.go:275] no items to output this cycle E1121 13:35:46.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:35:46.412252 3798194 memory.go:184] no items to output this cycle I1121 13:35:46.412401 3798194 cpu.go:275] no items to output this cycle E1121 13:35:56.411932 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:35:56.411966 3798194 memory.go:191] Add success. W1121 13:35:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:35:56.412009 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:35:56.412012 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:35:56.412338 3798194 cpu.go:282] Add success. I1121 13:35:56.441634 3798194 event_worker.go:152] Polling the log file for events... I1121 13:35:56.444593 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:35:56.445071 3798194 net.go:642] Add success. I1121 13:35:56.448920 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795237556, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 13:35:56.449040 3798194 net.go:778] primary dev: ETH0 I1121 13:35:56.449054 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:35:56.449068 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:35:56.456392 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 I1121 13:35:57.426279 3798194 disk_info.go:125] begin check local disk info of client I1121 13:35:57.429789 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:35:57.429798 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003865c0 0xc000386600] E1121 13:35:57.449577 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 13:35:57.449588 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:35:57.449786 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:35:57.449796 3798194 disk_worker.go:723] disk space is not compliant W1121 13:35:57.449799 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:35:57.457866 3798194 disk_worker.go:509] system disk:vda1 I1121 13:35:57.457909 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 13:35:58.447791 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 13:35:58.447807 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 13:35:59.448626 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 13:35:59.449698 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:35:59.449751 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:35:59.449770 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:35:59.471306 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:36:06.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:36:06.412261 3798194 memory.go:184] no items to output this cycle I1121 13:36:06.412407 3798194 cpu.go:275] no items to output this cycle E1121 13:36:16.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:36:16.411987 3798194 memory.go:184] no items to output this cycle I1121 13:36:16.412159 3798194 cpu.go:275] no items to output this cycle I1121 13:36:17.476572 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:36:17.476581 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:36:26.411220 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:36:26.411252 3798194 memory.go:191] Add success. I1121 13:36:26.411598 3798194 cpu.go:282] Add success. I1121 13:36:26.442206 3798194 net.go:642] Add success. I1121 13:36:26.446428 3798194 net.go:778] primary dev: ETH0 I1121 13:36:26.446442 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:36:26.446456 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:36:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:36:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:36:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:36:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:36:36.412251 3798194 memory.go:184] no items to output this cycle I1121 13:36:36.412396 3798194 cpu.go:275] no items to output this cycle E1121 13:36:46.413224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:36:46.413246 3798194 memory.go:184] no items to output this cycle I1121 13:36:46.413397 3798194 cpu.go:275] no items to output this cycle E1121 13:36:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:36:56.411949 3798194 memory.go:191] Add success. W1121 13:36:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:36:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:36:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:36:56.412515 3798194 cpu.go:282] Add success. I1121 13:36:56.440975 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:36:56.441563 3798194 net.go:642] Add success. I1121 13:36:56.445162 3798194 net.go:778] primary dev: ETH0 I1121 13:36:56.445177 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:36:56.445191 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:36:57.429882 3798194 disk_info.go:125] begin check local disk info of client I1121 13:36:57.433466 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:36:57.433475 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071880 0xc0000718c0] I1121 13:36:57.450245 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:36:57.450405 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:36:57.450456 3798194 disk_worker.go:723] disk space is not compliant W1121 13:36:57.450459 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:36:57.458499 3798194 disk_worker.go:509] system disk:vda1 I1121 13:36:57.458537 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:36:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:36:59.450107 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:36:59.450196 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:36:59.450224 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:36:59.470669 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:37:06.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:37:06.411984 3798194 memory.go:184] no items to output this cycle I1121 13:37:06.412140 3798194 cpu.go:275] no items to output this cycle E1121 13:37:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:37:16.413257 3798194 memory.go:184] no items to output this cycle I1121 13:37:16.413408 3798194 cpu.go:275] no items to output this cycle E1121 13:37:26.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:37:26.411956 3798194 memory.go:191] Add success. I1121 13:37:26.412319 3798194 cpu.go:282] Add success. I1121 13:37:26.441960 3798194 net.go:642] Add success. I1121 13:37:26.446396 3798194 net.go:778] primary dev: ETH0 I1121 13:37:26.446411 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:37:26.446424 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:37:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:37:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:37:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:37:36.412591 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:37:36.412612 3798194 memory.go:184] no items to output this cycle I1121 13:37:36.412756 3798194 cpu.go:275] no items to output this cycle E1121 13:37:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:37:46.413265 3798194 memory.go:184] no items to output this cycle I1121 13:37:46.413413 3798194 cpu.go:275] no items to output this cycle E1121 13:37:56.411989 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:37:56.412021 3798194 memory.go:191] Add success. W1121 13:37:56.412055 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:37:56.412070 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:37:56.412073 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:37:56.412423 3798194 cpu.go:282] Add success. I1121 13:37:56.441897 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:37:56.442560 3798194 net.go:642] Add success. I1121 13:37:56.446756 3798194 net.go:778] primary dev: ETH0 I1121 13:37:56.446771 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:37:56.446786 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:37:57.434311 3798194 disk_info.go:125] begin check local disk info of client I1121 13:37:57.437882 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:37:57.437891 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041c3c0 0xc00041c400] I1121 13:37:57.449710 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:37:57.449915 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:37:57.449926 3798194 disk_worker.go:723] disk space is not compliant W1121 13:37:57.449929 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:37:57.458300 3798194 disk_worker.go:509] system disk:vda1 I1121 13:37:57.458344 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:37:58.446389 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:37:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:37:59.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:37:59.450492 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:37:59.470994 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:38:06.413036 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:38:06.413059 3798194 memory.go:184] no items to output this cycle I1121 13:38:06.413412 3798194 cpu.go:275] no items to output this cycle E1121 13:38:16.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:38:16.412257 3798194 memory.go:184] no items to output this cycle I1121 13:38:16.412402 3798194 cpu.go:275] no items to output this cycle E1121 13:38:26.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:38:26.412892 3798194 memory.go:191] Add success. I1121 13:38:26.417163 3798194 cpu.go:282] Add success. I1121 13:38:26.441611 3798194 net.go:642] Add success. I1121 13:38:26.442939 3798194 net.go:778] primary dev: ETH0 I1121 13:38:26.442953 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:38:26.442969 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:38:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:38:29.450465 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:38:29.450494 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:38:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:38:36.411975 3798194 memory.go:184] no items to output this cycle I1121 13:38:36.412122 3798194 cpu.go:275] no items to output this cycle E1121 13:38:46.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:38:46.412263 3798194 memory.go:184] no items to output this cycle I1121 13:38:46.412412 3798194 cpu.go:275] no items to output this cycle E1121 13:38:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:38:56.411950 3798194 memory.go:191] Add success. W1121 13:38:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:38:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:38:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:38:56.412323 3798194 cpu.go:282] Add success. I1121 13:38:56.441809 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:38:56.442444 3798194 net.go:642] Add success. I1121 13:38:56.446628 3798194 net.go:778] primary dev: ETH0 I1121 13:38:56.446652 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:38:56.446666 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:38:57.438339 3798194 disk_info.go:125] begin check local disk info of client I1121 13:38:57.441852 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:38:57.441860 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000343600 0xc000343640] I1121 13:38:57.449615 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:38:57.449811 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:38:57.449824 3798194 disk_worker.go:723] disk space is not compliant W1121 13:38:57.449826 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:38:57.458020 3798194 disk_worker.go:509] system disk:vda1 I1121 13:38:57.458063 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:38:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:38:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:38:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:38:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:38:59.471657 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:39:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:39:06.412256 3798194 memory.go:184] no items to output this cycle I1121 13:39:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 13:39:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:39:16.412255 3798194 memory.go:184] no items to output this cycle I1121 13:39:16.412406 3798194 cpu.go:275] no items to output this cycle I1121 13:39:17.476736 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:39:17.476745 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:39:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:39:26.413268 3798194 memory.go:191] Add success. I1121 13:39:26.417502 3798194 cpu.go:282] Add success. I1121 13:39:26.441247 3798194 net.go:642] Add success. I1121 13:39:26.442718 3798194 net.go:778] primary dev: ETH0 I1121 13:39:26.442735 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:39:26.442748 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:39:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:39:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:39:29.450149 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:39:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:39:36.411979 3798194 memory.go:184] no items to output this cycle I1121 13:39:36.412357 3798194 cpu.go:275] no items to output this cycle E1121 13:39:46.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:39:46.411985 3798194 memory.go:184] no items to output this cycle I1121 13:39:46.412146 3798194 cpu.go:275] no items to output this cycle E1121 13:39:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:39:56.411955 3798194 memory.go:191] Add success. W1121 13:39:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:39:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:39:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:39:56.412376 3798194 cpu.go:282] Add success. I1121 13:39:56.441943 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:39:56.442549 3798194 net.go:642] Add success. I1121 13:39:56.446783 3798194 net.go:778] primary dev: ETH0 I1121 13:39:56.446797 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:39:56.446811 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:39:57.441934 3798194 disk_info.go:125] begin check local disk info of client I1121 13:39:57.454605 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:39:57.454613 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d9600 0xc0003d9640] I1121 13:39:57.454627 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:39:57.454817 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:39:57.454826 3798194 disk_worker.go:723] disk space is not compliant W1121 13:39:57.454829 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:39:57.463678 3798194 disk_worker.go:509] system disk:vda1 I1121 13:39:57.463727 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:39:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:39:59.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:39:59.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:39:59.450147 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:39:59.471659 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:40:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:40:06.411983 3798194 memory.go:184] no items to output this cycle I1121 13:40:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 13:40:16.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:40:16.412248 3798194 memory.go:184] no items to output this cycle I1121 13:40:16.412396 3798194 cpu.go:275] no items to output this cycle E1121 13:40:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:40:26.413268 3798194 memory.go:191] Add success. I1121 13:40:26.417540 3798194 cpu.go:282] Add success. I1121 13:40:26.442187 3798194 net.go:642] Add success. I1121 13:40:26.443830 3798194 net.go:778] primary dev: ETH0 I1121 13:40:26.443844 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:40:26.443859 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:40:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:40:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:40:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:40:36.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:40:36.411974 3798194 memory.go:184] no items to output this cycle I1121 13:40:36.412125 3798194 cpu.go:275] no items to output this cycle E1121 13:40:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:40:46.411970 3798194 memory.go:184] no items to output this cycle I1121 13:40:46.412132 3798194 cpu.go:275] no items to output this cycle E1121 13:40:56.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:40:56.411946 3798194 memory.go:191] Add success. W1121 13:40:56.411971 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:40:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:40:56.411985 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:40:56.412334 3798194 cpu.go:282] Add success. I1121 13:40:56.442211 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:40:56.442708 3798194 net.go:642] Add success. I1121 13:40:56.451557 3798194 net.go:778] primary dev: ETH0 I1121 13:40:56.451574 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:40:56.451586 3798194 net.go:659] Add eni metrics success, count: 7 E1121 13:40:57.447943 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 13:40:57.447963 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:40:57.448209 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:40:57.448219 3798194 disk_worker.go:723] disk space is not compliant W1121 13:40:57.448222 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:40:57.450986 3798194 disk_worker.go:509] system disk:vda1 I1121 13:40:57.451025 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:40:57.458723 3798194 disk_info.go:125] begin check local disk info of client I1121 13:40:57.463388 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:40:57.463396 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000477c80 0xc000477cc0] E1121 13:40:58.447814 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 13:40:58.447829 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 13:40:59.448628 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 13:40:59.449659 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:40:59.449708 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:40:59.449736 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:40:59.471187 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:41:06.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:41:06.411991 3798194 memory.go:184] no items to output this cycle I1121 13:41:06.412142 3798194 cpu.go:275] no items to output this cycle E1121 13:41:16.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:41:16.413253 3798194 memory.go:184] no items to output this cycle I1121 13:41:16.413406 3798194 cpu.go:275] no items to output this cycle E1121 13:41:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:41:26.411957 3798194 memory.go:191] Add success. I1121 13:41:26.412312 3798194 cpu.go:282] Add success. I1121 13:41:26.441947 3798194 net.go:642] Add success. I1121 13:41:26.446166 3798194 net.go:778] primary dev: ETH0 I1121 13:41:26.446181 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:41:26.446195 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:41:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:41:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:41:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:41:36.413165 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:41:36.413189 3798194 memory.go:184] no items to output this cycle I1121 13:41:36.413331 3798194 cpu.go:275] no items to output this cycle E1121 13:41:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:41:46.411972 3798194 memory.go:184] no items to output this cycle I1121 13:41:46.412127 3798194 cpu.go:275] no items to output this cycle E1121 13:41:56.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:41:56.412885 3798194 memory.go:191] Add success. W1121 13:41:56.412912 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:41:56.412923 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:41:56.412927 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:41:56.417202 3798194 cpu.go:282] Add success. I1121 13:41:56.441601 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:41:56.442094 3798194 net.go:642] Add success. I1121 13:41:56.443602 3798194 net.go:778] primary dev: ETH0 I1121 13:41:56.443619 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:41:56.443631 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:41:57.450286 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:41:57.450487 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:41:57.450499 3798194 disk_worker.go:723] disk space is not compliant W1121 13:41:57.450502 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:41:57.458480 3798194 disk_worker.go:509] system disk:vda1 I1121 13:41:57.458521 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:41:57.464945 3798194 disk_info.go:125] begin check local disk info of client I1121 13:41:57.467718 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:41:57.467725 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f6f40 0xc0003f6f80] I1121 13:41:58.446133 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:41:59.450234 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:41:59.450315 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:41:59.450340 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:41:59.470843 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:42:06.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:42:06.412879 3798194 memory.go:184] no items to output this cycle I1121 13:42:06.413023 3798194 cpu.go:275] no items to output this cycle E1121 13:42:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:42:16.411976 3798194 memory.go:184] no items to output this cycle I1121 13:42:16.412337 3798194 cpu.go:275] no items to output this cycle I1121 13:42:17.476904 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:42:17.476913 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:42:26.412282 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:42:26.412324 3798194 memory.go:191] Add success. I1121 13:42:26.417705 3798194 cpu.go:282] Add success. I1121 13:42:26.442255 3798194 net.go:642] Add success. I1121 13:42:26.443853 3798194 net.go:778] primary dev: ETH0 I1121 13:42:26.443869 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:42:26.443884 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:42:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:42:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:42:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:42:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:42:36.413263 3798194 memory.go:184] no items to output this cycle I1121 13:42:36.413408 3798194 cpu.go:275] no items to output this cycle E1121 13:42:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:42:46.413253 3798194 memory.go:184] no items to output this cycle I1121 13:42:46.413395 3798194 cpu.go:275] no items to output this cycle E1121 13:42:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:42:56.411950 3798194 memory.go:191] Add success. W1121 13:42:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:42:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:42:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:42:56.412327 3798194 cpu.go:282] Add success. I1121 13:42:56.441754 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:42:56.442261 3798194 net.go:642] Add success. I1121 13:42:56.446256 3798194 net.go:778] primary dev: ETH0 I1121 13:42:56.446270 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:42:56.446284 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:42:57.450287 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:42:57.450494 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:42:57.450504 3798194 disk_worker.go:723] disk space is not compliant W1121 13:42:57.450507 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:42:57.458793 3798194 disk_worker.go:509] system disk:vda1 I1121 13:42:57.458833 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:42:57.467872 3798194 disk_info.go:125] begin check local disk info of client I1121 13:42:57.470592 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:42:57.470599 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002ebb80 0xc0002ebbc0] I1121 13:42:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:42:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:42:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:42:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:42:59.471672 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:43:06.412782 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:43:06.412807 3798194 memory.go:184] no items to output this cycle I1121 13:43:06.412956 3798194 cpu.go:275] no items to output this cycle E1121 13:43:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:43:16.411985 3798194 memory.go:184] no items to output this cycle I1121 13:43:16.412146 3798194 cpu.go:275] no items to output this cycle E1121 13:43:26.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:43:26.412261 3798194 memory.go:191] Add success. I1121 13:43:26.416432 3798194 cpu.go:282] Add success. I1121 13:43:26.442184 3798194 net.go:642] Add success. I1121 13:43:26.443515 3798194 net.go:778] primary dev: ETH0 I1121 13:43:26.443529 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:43:26.443543 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:43:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:43:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:43:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:43:36.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:43:36.412887 3798194 memory.go:184] no items to output this cycle I1121 13:43:36.413039 3798194 cpu.go:275] no items to output this cycle E1121 13:43:46.413054 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:43:46.413080 3798194 memory.go:184] no items to output this cycle I1121 13:43:46.413244 3798194 cpu.go:275] no items to output this cycle E1121 13:43:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:43:56.411947 3798194 memory.go:191] Add success. W1121 13:43:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:43:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:43:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:43:56.412343 3798194 cpu.go:282] Add success. I1121 13:43:56.441125 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:43:56.441598 3798194 net.go:642] Add success. I1121 13:43:56.445759 3798194 net.go:778] primary dev: ETH0 I1121 13:43:56.445773 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:43:56.445786 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:43:57.450291 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:43:57.450464 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:43:57.450521 3798194 disk_worker.go:723] disk space is not compliant W1121 13:43:57.450524 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:43:57.458564 3798194 disk_worker.go:509] system disk:vda1 I1121 13:43:57.458606 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:43:57.471490 3798194 disk_info.go:125] begin check local disk info of client I1121 13:43:57.474243 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:43:57.474250 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000367dc0 0xc000367e00] I1121 13:43:58.446398 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:43:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:43:59.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:43:59.450127 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:43:59.471692 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:44:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:44:06.411981 3798194 memory.go:184] no items to output this cycle I1121 13:44:06.412139 3798194 cpu.go:275] no items to output this cycle E1121 13:44:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:44:16.411984 3798194 memory.go:184] no items to output this cycle I1121 13:44:16.412142 3798194 cpu.go:275] no items to output this cycle E1121 13:44:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:44:26.411955 3798194 memory.go:191] Add success. I1121 13:44:26.412311 3798194 cpu.go:282] Add success. I1121 13:44:26.442041 3798194 net.go:642] Add success. I1121 13:44:26.446605 3798194 net.go:778] primary dev: ETH0 I1121 13:44:26.446620 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:44:26.446634 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:44:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:44:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:44:29.450496 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:44:36.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:44:36.411984 3798194 memory.go:184] no items to output this cycle I1121 13:44:36.412138 3798194 cpu.go:275] no items to output this cycle E1121 13:44:46.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:44:46.412264 3798194 memory.go:184] no items to output this cycle I1121 13:44:46.412406 3798194 cpu.go:275] no items to output this cycle E1121 13:44:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:44:56.411956 3798194 memory.go:191] Add success. W1121 13:44:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:44:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:44:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:44:56.412368 3798194 cpu.go:282] Add success. I1121 13:44:56.441833 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:44:56.442358 3798194 net.go:642] Add success. I1121 13:44:56.446703 3798194 net.go:778] primary dev: ETH0 I1121 13:44:56.446719 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:44:56.446731 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:44:57.450272 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:44:57.450439 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:44:57.450495 3798194 disk_worker.go:723] disk space is not compliant W1121 13:44:57.450499 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:44:57.459452 3798194 disk_worker.go:509] system disk:vda1 I1121 13:44:57.459495 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:44:57.474514 3798194 disk_info.go:125] begin check local disk info of client I1121 13:44:57.479003 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:44:57.479010 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004bf680 0xc0004bf6c0] I1121 13:44:58.445923 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:44:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:44:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:44:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:44:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:45:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:45:06.412261 3798194 memory.go:184] no items to output this cycle I1121 13:45:06.412411 3798194 cpu.go:275] no items to output this cycle E1121 13:45:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:45:16.411976 3798194 memory.go:184] no items to output this cycle I1121 13:45:16.412141 3798194 cpu.go:275] no items to output this cycle I1121 13:45:17.477078 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:45:17.477087 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:45:26.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:45:26.412270 3798194 memory.go:191] Add success. I1121 13:45:26.416692 3798194 cpu.go:282] Add success. I1121 13:45:26.441284 3798194 net.go:642] Add success. I1121 13:45:26.443082 3798194 net.go:778] primary dev: ETH0 I1121 13:45:26.443100 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:45:26.443126 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:45:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:45:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:45:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:45:36.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:45:36.412262 3798194 memory.go:184] no items to output this cycle I1121 13:45:36.412408 3798194 cpu.go:275] no items to output this cycle E1121 13:45:46.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:45:46.412252 3798194 memory.go:184] no items to output this cycle I1121 13:45:46.412406 3798194 cpu.go:275] no items to output this cycle E1121 13:45:56.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:45:56.413266 3798194 memory.go:191] Add success. W1121 13:45:56.413296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:45:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:45:56.413311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:45:56.417458 3798194 cpu.go:282] Add success. I1121 13:45:56.441963 3798194 event_worker.go:152] Polling the log file for events... I1121 13:45:56.442232 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:45:56.442731 3798194 net.go:642] Add success. I1121 13:45:56.445711 3798194 net.go:778] primary dev: ETH0 I1121 13:45:56.445726 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:45:56.445739 3798194 net.go:659] Add eni metrics success, count: 7 E1121 13:45:57.445178 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 13:45:57.445195 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:45:57.445400 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:45:57.445472 3798194 disk_worker.go:723] disk space is not compliant W1121 13:45:57.445476 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:45:57.447922 3798194 disk_worker.go:509] system disk:vda1 I1121 13:45:57.447963 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:45:57.479858 3798194 disk_info.go:125] begin check local disk info of client I1121 13:45:57.482833 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:45:57.482839 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c4b80 0xc0003c4bc0] E1121 13:45:58.447792 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 13:45:58.447807 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 13:45:59.448586 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 13:45:59.449657 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:45:59.449704 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:45:59.449729 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:45:59.471203 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:46:06.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:46:06.412264 3798194 memory.go:184] no items to output this cycle I1121 13:46:06.412415 3798194 cpu.go:275] no items to output this cycle E1121 13:46:16.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:46:16.411987 3798194 memory.go:184] no items to output this cycle I1121 13:46:16.412144 3798194 cpu.go:275] no items to output this cycle E1121 13:46:26.411933 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:46:26.411971 3798194 memory.go:191] Add success. I1121 13:46:26.412347 3798194 cpu.go:282] Add success. I1121 13:46:26.441952 3798194 net.go:642] Add success. I1121 13:46:26.446012 3798194 net.go:778] primary dev: ETH0 I1121 13:46:26.446027 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:46:26.446040 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:46:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:46:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:46:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:46:36.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:46:36.412260 3798194 memory.go:184] no items to output this cycle I1121 13:46:36.412413 3798194 cpu.go:275] no items to output this cycle E1121 13:46:46.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:46:46.411970 3798194 memory.go:184] no items to output this cycle I1121 13:46:46.412143 3798194 cpu.go:275] no items to output this cycle E1121 13:46:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:46:56.411954 3798194 memory.go:191] Add success. W1121 13:46:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:46:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:46:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:46:56.412353 3798194 cpu.go:282] Add success. I1121 13:46:56.441814 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:46:56.442432 3798194 net.go:642] Add success. I1121 13:46:56.446830 3798194 net.go:778] primary dev: ETH0 I1121 13:46:56.446847 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:46:56.446859 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:46:57.450314 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:46:57.450531 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:46:57.450542 3798194 disk_worker.go:723] disk space is not compliant W1121 13:46:57.450546 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:46:57.458546 3798194 disk_worker.go:509] system disk:vda1 I1121 13:46:57.458587 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:46:57.483700 3798194 disk_info.go:125] begin check local disk info of client I1121 13:46:57.486487 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:46:57.486494 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000322b40 0xc000322b80] I1121 13:46:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:46:59.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:46:59.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:46:59.450507 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:46:59.471013 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:47:06.412953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:47:06.412978 3798194 memory.go:184] no items to output this cycle I1121 13:47:06.413142 3798194 cpu.go:275] no items to output this cycle E1121 13:47:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:47:16.411980 3798194 memory.go:184] no items to output this cycle I1121 13:47:16.412139 3798194 cpu.go:275] no items to output this cycle E1121 13:47:26.411934 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:47:26.411969 3798194 memory.go:191] Add success. I1121 13:47:26.412341 3798194 cpu.go:282] Add success. I1121 13:47:26.442060 3798194 net.go:642] Add success. I1121 13:47:26.446176 3798194 net.go:778] primary dev: ETH0 I1121 13:47:26.446194 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:47:26.446208 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:47:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:47:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:47:29.450498 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:47:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:47:36.413261 3798194 memory.go:184] no items to output this cycle I1121 13:47:36.413406 3798194 cpu.go:275] no items to output this cycle E1121 13:47:46.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:47:46.413264 3798194 memory.go:184] no items to output this cycle I1121 13:47:46.413412 3798194 cpu.go:275] no items to output this cycle E1121 13:47:56.412051 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:47:56.412162 3798194 memory.go:191] Add success. W1121 13:47:56.412198 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:47:56.412211 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:47:56.412215 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:47:56.412885 3798194 cpu.go:282] Add success. I1121 13:47:56.440961 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:47:56.441564 3798194 net.go:642] Add success. I1121 13:47:56.445710 3798194 net.go:778] primary dev: ETH0 I1121 13:47:56.445725 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:47:56.445738 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:47:57.453447 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:47:57.453756 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:47:57.453782 3798194 disk_worker.go:723] disk space is not compliant W1121 13:47:57.453787 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:47:57.462307 3798194 disk_worker.go:509] system disk:vda1 I1121 13:47:57.462347 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:47:57.486552 3798194 disk_info.go:125] begin check local disk info of client I1121 13:47:57.489582 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:47:57.489590 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e380 0xc00030e3c0] I1121 13:47:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:47:59.450409 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:47:59.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:47:59.450510 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:47:59.471024 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:48:06.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:48:06.413262 3798194 memory.go:184] no items to output this cycle I1121 13:48:06.413405 3798194 cpu.go:275] no items to output this cycle E1121 13:48:16.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:48:16.413266 3798194 memory.go:184] no items to output this cycle I1121 13:48:16.413413 3798194 cpu.go:275] no items to output this cycle I1121 13:48:17.477198 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:48:17.477206 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:48:26.414689 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:48:26.414725 3798194 memory.go:191] Add success. I1121 13:48:26.415049 3798194 cpu.go:282] Add success. I1121 13:48:26.441628 3798194 net.go:642] Add success. I1121 13:48:26.445900 3798194 net.go:778] primary dev: ETH0 I1121 13:48:26.445916 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:48:26.445929 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:48:29.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:48:29.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:48:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:48:36.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:48:36.411973 3798194 memory.go:184] no items to output this cycle I1121 13:48:36.412130 3798194 cpu.go:275] no items to output this cycle E1121 13:48:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:48:46.413254 3798194 memory.go:184] no items to output this cycle I1121 13:48:46.413401 3798194 cpu.go:275] no items to output this cycle E1121 13:48:56.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:48:56.412272 3798194 memory.go:191] Add success. W1121 13:48:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:48:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:48:56.412314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:48:56.416668 3798194 cpu.go:282] Add success. I1121 13:48:56.440952 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:48:56.441539 3798194 net.go:642] Add success. I1121 13:48:56.443276 3798194 net.go:778] primary dev: ETH0 I1121 13:48:56.443296 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:48:56.443309 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:48:57.450228 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:48:57.450428 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:48:57.450439 3798194 disk_worker.go:723] disk space is not compliant W1121 13:48:57.450442 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:48:57.458454 3798194 disk_worker.go:509] system disk:vda1 I1121 13:48:57.458496 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:48:57.490593 3798194 disk_info.go:125] begin check local disk info of client I1121 13:48:57.493452 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:48:57.493461 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046c000 0xc00046c040] I1121 13:48:58.446011 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:48:59.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:48:59.450122 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:48:59.450148 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:48:59.471701 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:49:06.412083 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:49:06.412186 3798194 memory.go:184] no items to output this cycle I1121 13:49:06.412671 3798194 cpu.go:275] no items to output this cycle E1121 13:49:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:49:16.411976 3798194 memory.go:184] no items to output this cycle I1121 13:49:16.412151 3798194 cpu.go:275] no items to output this cycle E1121 13:49:26.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:49:26.412889 3798194 memory.go:191] Add success. I1121 13:49:26.417202 3798194 cpu.go:282] Add success. I1121 13:49:26.441664 3798194 net.go:642] Add success. I1121 13:49:26.443178 3798194 net.go:778] primary dev: ETH0 I1121 13:49:26.443195 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:49:26.443207 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:49:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:49:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:49:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:49:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:49:36.413260 3798194 memory.go:184] no items to output this cycle I1121 13:49:36.413403 3798194 cpu.go:275] no items to output this cycle E1121 13:49:46.412965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:49:46.412988 3798194 memory.go:184] no items to output this cycle I1121 13:49:46.413159 3798194 cpu.go:275] no items to output this cycle E1121 13:49:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:49:56.411950 3798194 memory.go:191] Add success. W1121 13:49:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:49:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:49:56.412000 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:49:56.412347 3798194 cpu.go:282] Add success. I1121 13:49:56.441814 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:49:56.442319 3798194 net.go:642] Add success. I1121 13:49:56.446196 3798194 net.go:778] primary dev: ETH0 I1121 13:49:56.446211 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:49:56.446224 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:49:57.450767 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:49:57.450984 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:49:57.450996 3798194 disk_worker.go:723] disk space is not compliant W1121 13:49:57.451000 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:49:57.459148 3798194 disk_worker.go:509] system disk:vda1 I1121 13:49:57.459190 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:49:57.494230 3798194 disk_info.go:125] begin check local disk info of client I1121 13:49:57.497036 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:49:57.497043 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00038d240 0xc00038d280] I1121 13:49:58.445881 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:49:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:49:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:49:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:49:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:50:06.413101 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:50:06.413145 3798194 memory.go:184] no items to output this cycle I1121 13:50:06.413296 3798194 cpu.go:275] no items to output this cycle E1121 13:50:16.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:50:16.412258 3798194 memory.go:184] no items to output this cycle I1121 13:50:16.412403 3798194 cpu.go:275] no items to output this cycle E1121 13:50:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:50:26.411946 3798194 memory.go:191] Add success. I1121 13:50:26.412505 3798194 cpu.go:282] Add success. I1121 13:50:26.441316 3798194 net.go:642] Add success. I1121 13:50:26.445524 3798194 net.go:778] primary dev: ETH0 I1121 13:50:26.445539 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:50:26.445553 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:50:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:50:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:50:29.450149 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:50:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:50:36.412250 3798194 memory.go:184] no items to output this cycle I1121 13:50:36.412394 3798194 cpu.go:275] no items to output this cycle E1121 13:50:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:50:46.411973 3798194 memory.go:184] no items to output this cycle I1121 13:50:46.412132 3798194 cpu.go:275] no items to output this cycle E1121 13:50:56.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:50:56.412260 3798194 memory.go:191] Add success. W1121 13:50:56.412288 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:50:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:50:56.412304 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:50:56.416846 3798194 cpu.go:282] Add success. I1121 13:50:56.443909 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:50:56.444434 3798194 net.go:642] Add success. I1121 13:50:56.446036 3798194 net.go:778] primary dev: ETH0 I1121 13:50:56.446051 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:50:56.446063 3798194 net.go:659] Add eni metrics success, count: 7 E1121 13:50:57.450250 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 13:50:57.450264 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:50:57.450472 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:50:57.450483 3798194 disk_worker.go:723] disk space is not compliant W1121 13:50:57.450486 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:50:57.458557 3798194 disk_worker.go:509] system disk:vda1 I1121 13:50:57.458601 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:50:57.497604 3798194 disk_info.go:125] begin check local disk info of client I1121 13:50:57.500375 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:50:57.500383 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000220800 0xc000220840] E1121 13:50:58.447787 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 13:50:58.447802 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:50:59.449889 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:50:59.449941 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:50:59.449959 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:50:59.450080 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 13:50:59.471560 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:51:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:51:06.412257 3798194 memory.go:184] no items to output this cycle I1121 13:51:06.412401 3798194 cpu.go:275] no items to output this cycle E1121 13:51:16.412625 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:51:16.412648 3798194 memory.go:184] no items to output this cycle I1121 13:51:16.412791 3798194 cpu.go:275] no items to output this cycle I1121 13:51:17.478073 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:51:17.478082 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:51:26.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:51:26.412885 3798194 memory.go:191] Add success. I1121 13:51:26.416947 3798194 cpu.go:282] Add success. I1121 13:51:26.441494 3798194 net.go:642] Add success. I1121 13:51:26.442993 3798194 net.go:778] primary dev: ETH0 I1121 13:51:26.443007 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:51:26.443020 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:51:29.450386 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:51:29.450463 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:51:29.450490 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:51:36.412255 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:51:36.412277 3798194 memory.go:184] no items to output this cycle I1121 13:51:36.412429 3798194 cpu.go:275] no items to output this cycle E1121 13:51:46.412253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:51:46.412282 3798194 memory.go:184] no items to output this cycle I1121 13:51:46.412487 3798194 cpu.go:275] no items to output this cycle E1121 13:51:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:51:56.411963 3798194 memory.go:191] Add success. W1121 13:51:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:51:56.412010 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:51:56.412013 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:51:56.412371 3798194 cpu.go:282] Add success. I1121 13:51:56.441829 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:51:56.442503 3798194 net.go:642] Add success. I1121 13:51:56.446600 3798194 net.go:778] primary dev: ETH0 I1121 13:51:56.446614 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:51:56.446626 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:51:57.450274 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:51:57.450439 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:51:57.450493 3798194 disk_worker.go:723] disk space is not compliant W1121 13:51:57.450496 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:51:57.458613 3798194 disk_worker.go:509] system disk:vda1 I1121 13:51:57.458659 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:51:57.500975 3798194 disk_info.go:125] begin check local disk info of client I1121 13:51:57.504309 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:51:57.504319 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000220a80 0xc000220ac0] I1121 13:51:58.446119 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:51:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:51:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:51:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:51:59.471698 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:52:06.412777 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:52:06.412798 3798194 memory.go:184] no items to output this cycle I1121 13:52:06.412951 3798194 cpu.go:275] no items to output this cycle E1121 13:52:16.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:52:16.412255 3798194 memory.go:184] no items to output this cycle I1121 13:52:16.412404 3798194 cpu.go:275] no items to output this cycle E1121 13:52:26.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:52:26.413278 3798194 memory.go:191] Add success. I1121 13:52:26.417436 3798194 cpu.go:282] Add success. I1121 13:52:26.441967 3798194 net.go:642] Add success. I1121 13:52:26.443475 3798194 net.go:778] primary dev: ETH0 I1121 13:52:26.443489 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:52:26.443503 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:52:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:52:29.450124 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:52:29.450155 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:52:36.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:52:36.411988 3798194 memory.go:184] no items to output this cycle I1121 13:52:36.412155 3798194 cpu.go:275] no items to output this cycle E1121 13:52:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:52:46.411989 3798194 memory.go:184] no items to output this cycle I1121 13:52:46.412148 3798194 cpu.go:275] no items to output this cycle E1121 13:52:56.411245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:52:56.411280 3798194 memory.go:191] Add success. W1121 13:52:56.411313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:52:56.411325 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:52:56.411329 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:52:56.416090 3798194 cpu.go:282] Add success. I1121 13:52:56.441574 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:52:56.442445 3798194 net.go:642] Add success. I1121 13:52:56.444248 3798194 net.go:778] primary dev: ETH0 I1121 13:52:56.444264 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:52:56.444282 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:52:57.450175 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:52:57.450399 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:52:57.450410 3798194 disk_worker.go:723] disk space is not compliant W1121 13:52:57.450414 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:52:57.459885 3798194 disk_worker.go:509] system disk:vda1 I1121 13:52:57.459927 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:52:57.505007 3798194 disk_info.go:125] begin check local disk info of client I1121 13:52:57.507825 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:52:57.507832 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e380 0xc00030e3c0] I1121 13:52:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:52:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:52:59.450467 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:52:59.450492 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:52:59.470996 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:53:06.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:53:06.411977 3798194 memory.go:184] no items to output this cycle I1121 13:53:06.412137 3798194 cpu.go:275] no items to output this cycle E1121 13:53:16.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:53:16.413269 3798194 memory.go:184] no items to output this cycle I1121 13:53:16.413416 3798194 cpu.go:275] no items to output this cycle E1121 13:53:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:53:26.411955 3798194 memory.go:191] Add success. I1121 13:53:26.412318 3798194 cpu.go:282] Add success. I1121 13:53:26.441942 3798194 net.go:642] Add success. I1121 13:53:26.445977 3798194 net.go:778] primary dev: ETH0 I1121 13:53:26.445993 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:53:26.446006 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:53:29.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:53:29.450500 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:53:29.450530 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:53:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:53:36.412259 3798194 memory.go:184] no items to output this cycle I1121 13:53:36.412409 3798194 cpu.go:275] no items to output this cycle E1121 13:53:46.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:53:46.412271 3798194 memory.go:184] no items to output this cycle I1121 13:53:46.412431 3798194 cpu.go:275] no items to output this cycle E1121 13:53:56.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:53:56.412896 3798194 memory.go:191] Add success. W1121 13:53:56.412930 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:53:56.412941 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:53:56.412944 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:53:56.417069 3798194 cpu.go:282] Add success. I1121 13:53:56.441437 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:53:56.441941 3798194 net.go:642] Add success. I1121 13:53:56.443607 3798194 net.go:778] primary dev: ETH0 I1121 13:53:56.443622 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:53:56.443634 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:53:57.450639 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:53:57.450871 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:53:57.450882 3798194 disk_worker.go:723] disk space is not compliant W1121 13:53:57.450885 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:53:57.463536 3798194 disk_worker.go:509] system disk:vda1 I1121 13:53:57.463575 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:53:57.507878 3798194 disk_info.go:125] begin check local disk info of client I1121 13:53:57.510736 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:53:57.510755 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00039b740 0xc00039b780] I1121 13:53:58.446372 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:53:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:53:59.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:53:59.450494 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:53:59.471010 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:54:06.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:54:06.413269 3798194 memory.go:184] no items to output this cycle I1121 13:54:06.413416 3798194 cpu.go:275] no items to output this cycle E1121 13:54:16.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:54:16.412265 3798194 memory.go:184] no items to output this cycle I1121 13:54:16.412410 3798194 cpu.go:275] no items to output this cycle I1121 13:54:17.478199 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:54:17.478207 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:54:26.411221 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:54:26.411257 3798194 memory.go:191] Add success. I1121 13:54:26.411567 3798194 cpu.go:282] Add success. I1121 13:54:26.442185 3798194 net.go:642] Add success. I1121 13:54:26.446582 3798194 net.go:778] primary dev: ETH0 I1121 13:54:26.446597 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:54:26.446609 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:54:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:54:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:54:29.450502 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:54:36.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:54:36.412260 3798194 memory.go:184] no items to output this cycle I1121 13:54:36.412406 3798194 cpu.go:275] no items to output this cycle E1121 13:54:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:54:46.411983 3798194 memory.go:184] no items to output this cycle I1121 13:54:46.412147 3798194 cpu.go:275] no items to output this cycle E1121 13:54:56.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:54:56.412891 3798194 memory.go:191] Add success. W1121 13:54:56.412924 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:54:56.412937 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:54:56.412940 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:54:56.417477 3798194 cpu.go:282] Add success. I1121 13:54:56.441870 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:54:56.442462 3798194 net.go:642] Add success. I1121 13:54:56.443986 3798194 net.go:778] primary dev: ETH0 I1121 13:54:56.444001 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:54:56.444012 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:54:57.450261 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:54:57.450428 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:54:57.450487 3798194 disk_worker.go:723] disk space is not compliant W1121 13:54:57.450490 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:54:57.458591 3798194 disk_worker.go:509] system disk:vda1 I1121 13:54:57.458632 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:54:57.510796 3798194 disk_info.go:125] begin check local disk info of client I1121 13:54:57.513697 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:54:57.513704 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004261c0 0xc000426200] I1121 13:54:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:54:59.450409 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:54:59.450494 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:54:59.450521 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:54:59.471050 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:55:06.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:55:06.411972 3798194 memory.go:184] no items to output this cycle I1121 13:55:06.412122 3798194 cpu.go:275] no items to output this cycle E1121 13:55:16.412500 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:55:16.412521 3798194 memory.go:184] no items to output this cycle I1121 13:55:16.412808 3798194 cpu.go:275] no items to output this cycle E1121 13:55:26.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:55:26.411960 3798194 memory.go:191] Add success. I1121 13:55:26.412327 3798194 cpu.go:282] Add success. I1121 13:55:26.441949 3798194 net.go:642] Add success. I1121 13:55:26.445849 3798194 net.go:778] primary dev: ETH0 I1121 13:55:26.445865 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:55:26.445888 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:55:29.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:55:29.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:55:29.450152 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:55:36.412753 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:55:36.412775 3798194 memory.go:184] no items to output this cycle I1121 13:55:36.412936 3798194 cpu.go:275] no items to output this cycle E1121 13:55:46.415245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:55:46.415278 3798194 memory.go:184] no items to output this cycle I1121 13:55:46.415422 3798194 cpu.go:275] no items to output this cycle E1121 13:55:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:55:56.411946 3798194 memory.go:191] Add success. W1121 13:55:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:55:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:55:56.411993 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:55:56.412319 3798194 cpu.go:282] Add success. I1121 13:55:56.441534 3798194 event_worker.go:152] Polling the log file for events... I1121 13:55:56.444402 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:55:56.444910 3798194 net.go:642] Add success. I1121 13:55:56.449132 3798194 net.go:778] primary dev: ETH0 I1121 13:55:56.449147 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:55:56.449159 3798194 net.go:659] Add eni metrics success, count: 7 E1121 13:55:57.449964 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 13:55:57.449978 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:55:57.450162 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:55:57.450216 3798194 disk_worker.go:723] disk space is not compliant W1121 13:55:57.450219 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:55:57.458405 3798194 disk_worker.go:509] system disk:vda1 I1121 13:55:57.458447 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:55:57.514526 3798194 disk_info.go:125] begin check local disk info of client I1121 13:55:57.517489 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:55:57.517497 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000343bc0 0xc000343c00] E1121 13:55:58.448191 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 13:55:58.448206 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 13:55:59.448570 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 13:55:59.449645 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:55:59.449693 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:55:59.449712 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:55:59.471181 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:56:06.412844 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:56:06.412867 3798194 memory.go:184] no items to output this cycle I1121 13:56:06.413007 3798194 cpu.go:275] no items to output this cycle E1121 13:56:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:56:16.413263 3798194 memory.go:184] no items to output this cycle I1121 13:56:16.413409 3798194 cpu.go:275] no items to output this cycle E1121 13:56:26.411938 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:56:26.411972 3798194 memory.go:191] Add success. I1121 13:56:26.412304 3798194 cpu.go:282] Add success. I1121 13:56:26.441916 3798194 net.go:642] Add success. I1121 13:56:26.445976 3798194 net.go:778] primary dev: ETH0 I1121 13:56:26.445992 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:56:26.446005 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:56:29.450008 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:56:29.450088 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:56:29.450129 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:56:36.413163 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:56:36.413185 3798194 memory.go:184] no items to output this cycle I1121 13:56:36.413340 3798194 cpu.go:275] no items to output this cycle E1121 13:56:46.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:56:46.413272 3798194 memory.go:184] no items to output this cycle I1121 13:56:46.413419 3798194 cpu.go:275] no items to output this cycle E1121 13:56:56.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:56:56.413278 3798194 memory.go:191] Add success. W1121 13:56:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:56:56.413322 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:56:56.413325 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:56:56.418254 3798194 cpu.go:282] Add success. I1121 13:56:56.441665 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:56:56.442199 3798194 net.go:642] Add success. I1121 13:56:56.443886 3798194 net.go:778] primary dev: ETH0 I1121 13:56:56.443902 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:56:56.443915 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:56:57.445147 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:56:57.445353 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:56:57.445430 3798194 disk_worker.go:723] disk space is not compliant W1121 13:56:57.445434 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:56:57.447531 3798194 disk_worker.go:509] system disk:vda1 I1121 13:56:57.447570 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:56:57.517585 3798194 disk_info.go:125] begin check local disk info of client I1121 13:56:57.520594 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:56:57.520601 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000343e80 0xc000343ec0] I1121 13:56:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:56:59.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:56:59.450136 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:56:59.450162 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:56:59.471645 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:57:06.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:57:06.412880 3798194 memory.go:184] no items to output this cycle I1121 13:57:06.413026 3798194 cpu.go:275] no items to output this cycle E1121 13:57:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:57:16.411984 3798194 memory.go:184] no items to output this cycle I1121 13:57:16.412148 3798194 cpu.go:275] no items to output this cycle I1121 13:57:17.478363 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 13:57:17.478372 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 13:57:26.412253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:57:26.412289 3798194 memory.go:191] Add success. I1121 13:57:26.416576 3798194 cpu.go:282] Add success. I1121 13:57:26.442259 3798194 net.go:642] Add success. I1121 13:57:26.443821 3798194 net.go:778] primary dev: ETH0 I1121 13:57:26.443837 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:57:26.443850 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:57:29.450136 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:57:29.450237 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:57:29.450277 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:57:36.413500 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:57:36.413523 3798194 memory.go:184] no items to output this cycle I1121 13:57:36.413668 3798194 cpu.go:275] no items to output this cycle E1121 13:57:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:57:46.411983 3798194 memory.go:184] no items to output this cycle I1121 13:57:46.412150 3798194 cpu.go:275] no items to output this cycle E1121 13:57:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:57:56.411957 3798194 memory.go:191] Add success. W1121 13:57:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:57:56.412001 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:57:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:57:56.412345 3798194 cpu.go:282] Add success. I1121 13:57:56.441917 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:57:56.442568 3798194 net.go:642] Add success. I1121 13:57:56.447075 3798194 net.go:778] primary dev: ETH0 I1121 13:57:56.447090 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:57:56.447103 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:57:57.450684 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:57:57.451062 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:57:57.451073 3798194 disk_worker.go:723] disk space is not compliant W1121 13:57:57.451076 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:57:57.461741 3798194 disk_worker.go:509] system disk:vda1 I1121 13:57:57.461783 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:57:57.520686 3798194 disk_info.go:125] begin check local disk info of client I1121 13:57:57.523553 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:57:57.523560 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047d540 0xc00047d580] I1121 13:57:58.446008 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:57:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:57:59.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:57:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:57:59.471663 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:58:06.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:58:06.412263 3798194 memory.go:184] no items to output this cycle I1121 13:58:06.412408 3798194 cpu.go:275] no items to output this cycle E1121 13:58:16.412846 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:58:16.412866 3798194 memory.go:184] no items to output this cycle I1121 13:58:16.413015 3798194 cpu.go:275] no items to output this cycle E1121 13:58:26.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:58:26.412897 3798194 memory.go:191] Add success. I1121 13:58:26.417216 3798194 cpu.go:282] Add success. I1121 13:58:26.441764 3798194 net.go:642] Add success. I1121 13:58:26.443207 3798194 net.go:778] primary dev: ETH0 I1121 13:58:26.443222 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:58:26.443235 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:58:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:58:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:58:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:58:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:58:36.413268 3798194 memory.go:184] no items to output this cycle I1121 13:58:36.413412 3798194 cpu.go:275] no items to output this cycle E1121 13:58:46.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:58:46.411990 3798194 memory.go:184] no items to output this cycle I1121 13:58:46.412150 3798194 cpu.go:275] no items to output this cycle E1121 13:58:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:58:56.411945 3798194 memory.go:191] Add success. W1121 13:58:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:58:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:58:56.412002 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:58:56.412351 3798194 cpu.go:282] Add success. I1121 13:58:56.441805 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:58:56.442421 3798194 net.go:642] Add success. I1121 13:58:56.446548 3798194 net.go:778] primary dev: ETH0 I1121 13:58:56.446562 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:58:56.446574 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:58:57.451025 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:58:57.451254 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:58:57.451265 3798194 disk_worker.go:723] disk space is not compliant W1121 13:58:57.451268 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:58:57.464932 3798194 disk_worker.go:509] system disk:vda1 I1121 13:58:57.464976 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:58:57.524202 3798194 disk_info.go:125] begin check local disk info of client I1121 13:58:57.527202 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:58:57.527209 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d8b40 0xc0003d8b80] I1121 13:58:58.445981 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:58:59.450565 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:58:59.450649 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:58:59.450678 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:58:59.471207 3798194 disk_local_worker.go:436] Get disk info: [] E1121 13:59:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:59:06.412258 3798194 memory.go:184] no items to output this cycle I1121 13:59:06.412416 3798194 cpu.go:275] no items to output this cycle E1121 13:59:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:59:16.411984 3798194 memory.go:184] no items to output this cycle I1121 13:59:16.412143 3798194 cpu.go:275] no items to output this cycle E1121 13:59:26.412246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:59:26.412284 3798194 memory.go:191] Add success. I1121 13:59:26.416439 3798194 cpu.go:282] Add success. I1121 13:59:26.441988 3798194 net.go:642] Add success. I1121 13:59:26.443470 3798194 net.go:778] primary dev: ETH0 I1121 13:59:26.443484 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:59:26.443497 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:59:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:59:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:59:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 13:59:36.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:59:36.411984 3798194 memory.go:184] no items to output this cycle I1121 13:59:36.412147 3798194 cpu.go:275] no items to output this cycle E1121 13:59:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:59:46.412256 3798194 memory.go:184] no items to output this cycle I1121 13:59:46.412404 3798194 cpu.go:275] no items to output this cycle E1121 13:59:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 13:59:56.411945 3798194 memory.go:191] Add success. W1121 13:59:56.411974 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 13:59:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 13:59:56.411990 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 13:59:56.412316 3798194 cpu.go:282] Add success. I1121 13:59:56.441759 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 13:59:56.442268 3798194 net.go:642] Add success. I1121 13:59:56.446536 3798194 net.go:778] primary dev: ETH0 I1121 13:59:56.446551 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 13:59:56.446564 3798194 net.go:659] Add eni metrics success, count: 7 I1121 13:59:57.450203 3798194 custom_config.go:106] query custom config with name: gpu W1121 13:59:57.450421 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 13:59:57.450431 3798194 disk_worker.go:723] disk space is not compliant W1121 13:59:57.450435 3798194 disk_worker.go:743] disk inode is not compliant I1121 13:59:57.458725 3798194 disk_worker.go:509] system disk:vda1 I1121 13:59:57.458769 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 13:59:57.527265 3798194 disk_info.go:125] begin check local disk info of client I1121 13:59:57.530568 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 13:59:57.530576 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030b300 0xc00030b340] I1121 13:59:58.446235 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 13:59:59.450199 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 13:59:59.450291 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 13:59:59.450319 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 13:59:59.470841 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:00:06.412762 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:00:06.412784 3798194 memory.go:184] no items to output this cycle I1121 14:00:06.412944 3798194 cpu.go:275] no items to output this cycle E1121 14:00:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:00:16.412260 3798194 memory.go:184] no items to output this cycle I1121 14:00:16.412411 3798194 cpu.go:275] no items to output this cycle I1121 14:00:17.478521 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:00:17.478534 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:00:26.411006 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:00:26.411038 3798194 memory.go:191] Add success. I1121 14:00:26.415583 3798194 cpu.go:282] Add success. I1121 14:00:26.442135 3798194 net.go:642] Add success. I1121 14:00:26.443513 3798194 net.go:778] primary dev: ETH0 I1121 14:00:26.443527 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:00:26.443541 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:00:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:00:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:00:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:00:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:00:36.413258 3798194 memory.go:184] no items to output this cycle I1121 14:00:36.413409 3798194 cpu.go:275] no items to output this cycle E1121 14:00:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:00:46.413258 3798194 memory.go:184] no items to output this cycle I1121 14:00:46.413403 3798194 cpu.go:275] no items to output this cycle E1121 14:00:56.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:00:56.413283 3798194 memory.go:191] Add success. W1121 14:00:56.413315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:00:56.413327 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:00:56.413330 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:00:56.417708 3798194 cpu.go:282] Add success. I1121 14:00:56.445701 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:00:56.446299 3798194 net.go:642] Add success. I1121 14:00:56.447774 3798194 net.go:778] primary dev: ETH0 I1121 14:00:56.447788 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:00:56.447802 3798194 net.go:659] Add eni metrics success, count: 7 E1121 14:00:57.450966 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 14:00:57.450983 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:00:57.451223 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:00:57.451304 3798194 disk_worker.go:723] disk space is not compliant W1121 14:00:57.451309 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:00:57.460517 3798194 disk_worker.go:509] system disk:vda1 I1121 14:00:57.460560 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:00:57.530667 3798194 disk_info.go:125] begin check local disk info of client I1121 14:00:57.533710 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:00:57.533718 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0cc0 0xc0002e0d00] E1121 14:00:58.448148 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 14:00:58.448163 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 14:00:59.448850 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 14:00:59.449925 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:00:59.449981 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:00:59.449999 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:00:59.471504 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:01:06.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:01:06.413261 3798194 memory.go:184] no items to output this cycle I1121 14:01:06.413407 3798194 cpu.go:275] no items to output this cycle E1121 14:01:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:01:16.413257 3798194 memory.go:184] no items to output this cycle I1121 14:01:16.413403 3798194 cpu.go:275] no items to output this cycle E1121 14:01:26.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:01:26.413277 3798194 memory.go:191] Add success. I1121 14:01:26.417690 3798194 cpu.go:282] Add success. I1121 14:01:26.442263 3798194 net.go:642] Add success. I1121 14:01:26.444020 3798194 net.go:778] primary dev: ETH0 I1121 14:01:26.444034 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:01:26.444057 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:01:29.450702 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:01:29.450769 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:01:29.450795 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:01:36.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:01:36.411989 3798194 memory.go:184] no items to output this cycle I1121 14:01:36.412151 3798194 cpu.go:275] no items to output this cycle E1121 14:01:46.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:01:46.413253 3798194 memory.go:184] no items to output this cycle I1121 14:01:46.413398 3798194 cpu.go:275] no items to output this cycle E1121 14:01:56.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:01:56.412891 3798194 memory.go:191] Add success. W1121 14:01:56.412925 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:01:56.412937 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:01:56.412940 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:01:56.417482 3798194 cpu.go:282] Add success. I1121 14:01:56.441902 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:01:56.442508 3798194 net.go:642] Add success. I1121 14:01:56.444053 3798194 net.go:778] primary dev: ETH0 I1121 14:01:56.444067 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:01:56.444081 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:01:57.450685 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:01:57.450887 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:01:57.450897 3798194 disk_worker.go:723] disk space is not compliant W1121 14:01:57.450901 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:01:57.458978 3798194 disk_worker.go:509] system disk:vda1 I1121 14:01:57.459021 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:01:57.533811 3798194 disk_info.go:125] begin check local disk info of client I1121 14:01:57.537018 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:01:57.537026 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004700c0 0xc000470100] I1121 14:01:58.446680 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:01:59.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:01:59.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:01:59.450504 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:01:59.471006 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:02:06.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:02:06.413255 3798194 memory.go:184] no items to output this cycle I1121 14:02:06.413400 3798194 cpu.go:275] no items to output this cycle E1121 14:02:16.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:02:16.411989 3798194 memory.go:184] no items to output this cycle I1121 14:02:16.412160 3798194 cpu.go:275] no items to output this cycle E1121 14:02:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:02:26.411955 3798194 memory.go:191] Add success. I1121 14:02:26.412289 3798194 cpu.go:282] Add success. I1121 14:02:26.441891 3798194 net.go:642] Add success. I1121 14:02:26.445708 3798194 net.go:778] primary dev: ETH0 I1121 14:02:26.445724 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:02:26.445738 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:02:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:02:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:02:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:02:36.412781 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:02:36.412811 3798194 memory.go:184] no items to output this cycle I1121 14:02:36.412970 3798194 cpu.go:275] no items to output this cycle E1121 14:02:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:02:46.411983 3798194 memory.go:184] no items to output this cycle I1121 14:02:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 14:02:56.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:02:56.412270 3798194 memory.go:191] Add success. W1121 14:02:56.412304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:02:56.412315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:02:56.412318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:02:56.417214 3798194 cpu.go:282] Add success. I1121 14:02:56.441571 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:02:56.442055 3798194 net.go:642] Add success. I1121 14:02:56.443748 3798194 net.go:778] primary dev: ETH0 I1121 14:02:56.443763 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:02:56.443775 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:02:57.451037 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:02:57.451239 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:02:57.451296 3798194 disk_worker.go:723] disk space is not compliant W1121 14:02:57.451300 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:02:57.459741 3798194 disk_worker.go:509] system disk:vda1 I1121 14:02:57.459784 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:02:57.537969 3798194 disk_info.go:125] begin check local disk info of client I1121 14:02:57.541196 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:02:57.541203 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00038f6c0 0xc00038f700] I1121 14:02:58.446387 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:02:59.450488 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:02:59.450568 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:02:59.450595 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:02:59.471126 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:03:06.413284 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:03:06.413341 3798194 memory.go:184] no items to output this cycle I1121 14:03:06.413547 3798194 cpu.go:275] no items to output this cycle E1121 14:03:16.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:03:16.413263 3798194 memory.go:184] no items to output this cycle I1121 14:03:16.413410 3798194 cpu.go:275] no items to output this cycle I1121 14:03:17.479607 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:03:17.479616 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:03:26.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:03:26.412270 3798194 memory.go:191] Add success. I1121 14:03:26.416579 3798194 cpu.go:282] Add success. I1121 14:03:26.442177 3798194 net.go:642] Add success. I1121 14:03:26.443593 3798194 net.go:778] primary dev: ETH0 I1121 14:03:26.443607 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:03:26.443620 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:03:29.450238 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:03:29.450314 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:03:29.450340 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:03:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:03:36.411979 3798194 memory.go:184] no items to output this cycle I1121 14:03:36.412333 3798194 cpu.go:275] no items to output this cycle E1121 14:03:46.413610 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:03:46.413633 3798194 memory.go:184] no items to output this cycle I1121 14:03:46.413800 3798194 cpu.go:275] no items to output this cycle E1121 14:03:56.413258 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:03:56.413296 3798194 memory.go:191] Add success. W1121 14:03:56.413328 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:03:56.413339 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:03:56.413342 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:03:56.417917 3798194 cpu.go:282] Add success. I1121 14:03:56.441339 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:03:56.441831 3798194 net.go:642] Add success. I1121 14:03:56.443409 3798194 net.go:778] primary dev: ETH0 I1121 14:03:56.443424 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:03:56.443438 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:03:57.448002 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:03:57.448244 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:03:57.448307 3798194 disk_worker.go:723] disk space is not compliant W1121 14:03:57.448310 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:03:57.451173 3798194 disk_worker.go:509] system disk:vda1 I1121 14:03:57.451213 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:03:57.541818 3798194 disk_info.go:125] begin check local disk info of client I1121 14:03:57.544950 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:03:57.544957 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046ad00 0xc00046ad40] I1121 14:03:58.446061 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:03:59.450403 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:03:59.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:03:59.450506 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:03:59.471017 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:04:06.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:04:06.413274 3798194 memory.go:184] no items to output this cycle I1121 14:04:06.413422 3798194 cpu.go:275] no items to output this cycle E1121 14:04:16.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:04:16.412256 3798194 memory.go:184] no items to output this cycle I1121 14:04:16.412403 3798194 cpu.go:275] no items to output this cycle E1121 14:04:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:04:26.411956 3798194 memory.go:191] Add success. I1121 14:04:26.412376 3798194 cpu.go:282] Add success. I1121 14:04:26.441941 3798194 net.go:642] Add success. I1121 14:04:26.446021 3798194 net.go:778] primary dev: ETH0 I1121 14:04:26.446036 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:04:26.446051 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:04:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:04:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:04:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:04:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:04:36.412258 3798194 memory.go:184] no items to output this cycle I1121 14:04:36.412400 3798194 cpu.go:275] no items to output this cycle E1121 14:04:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:04:46.413258 3798194 memory.go:184] no items to output this cycle I1121 14:04:46.413407 3798194 cpu.go:275] no items to output this cycle E1121 14:04:56.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:04:56.412275 3798194 memory.go:191] Add success. W1121 14:04:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:04:56.412316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:04:56.412319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:04:56.416535 3798194 cpu.go:282] Add success. I1121 14:04:56.440898 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:04:56.441416 3798194 net.go:642] Add success. I1121 14:04:56.442933 3798194 net.go:778] primary dev: ETH0 I1121 14:04:56.442947 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:04:56.442961 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:04:57.448843 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:04:57.449082 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:04:57.449095 3798194 disk_worker.go:723] disk space is not compliant W1121 14:04:57.449098 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:04:57.451739 3798194 disk_worker.go:509] system disk:vda1 I1121 14:04:57.451776 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:04:57.545193 3798194 disk_info.go:125] begin check local disk info of client I1121 14:04:57.548265 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:04:57.548272 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f38c0 0xc0003f3900] I1121 14:04:58.445990 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:04:59.450492 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:04:59.450573 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:04:59.450598 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:04:59.471131 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:05:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:05:06.412262 3798194 memory.go:184] no items to output this cycle I1121 14:05:06.412411 3798194 cpu.go:275] no items to output this cycle E1121 14:05:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:05:16.413260 3798194 memory.go:184] no items to output this cycle I1121 14:05:16.413416 3798194 cpu.go:275] no items to output this cycle E1121 14:05:26.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:05:26.412896 3798194 memory.go:191] Add success. I1121 14:05:26.417554 3798194 cpu.go:282] Add success. I1121 14:05:26.442107 3798194 net.go:642] Add success. I1121 14:05:26.443633 3798194 net.go:778] primary dev: ETH0 I1121 14:05:26.443648 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:05:26.443661 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:05:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:05:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:05:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:05:36.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:05:36.412264 3798194 memory.go:184] no items to output this cycle I1121 14:05:36.412405 3798194 cpu.go:275] no items to output this cycle E1121 14:05:46.415299 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:05:46.415328 3798194 memory.go:184] no items to output this cycle I1121 14:05:46.415548 3798194 cpu.go:275] no items to output this cycle E1121 14:05:56.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:05:56.412274 3798194 memory.go:191] Add success. W1121 14:05:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:05:56.412315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:05:56.412318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:05:56.416573 3798194 cpu.go:282] Add success. I1121 14:05:56.441942 3798194 event_worker.go:152] Polling the log file for events... I1121 14:05:56.442337 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:05:56.442814 3798194 net.go:642] Add success. I1121 14:05:56.446097 3798194 net.go:778] primary dev: ETH0 I1121 14:05:56.446138 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:05:56.446152 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:05:56.446167 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795239356, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 14:05:56.451659 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 14:05:57.445285 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 14:05:57.445303 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:05:57.445510 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:05:57.445578 3798194 disk_worker.go:723] disk space is not compliant W1121 14:05:57.445582 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:05:57.448034 3798194 disk_worker.go:509] system disk:vda1 I1121 14:05:57.448077 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:05:57.549083 3798194 disk_info.go:125] begin check local disk info of client I1121 14:05:57.552154 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:05:57.552162 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b4e00 0xc0003b4e40] E1121 14:05:58.447775 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 14:05:58.447790 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 14:05:59.448997 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 14:05:59.450071 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:05:59.450131 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:05:59.450154 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:05:59.471682 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:06:06.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:06:06.412256 3798194 memory.go:184] no items to output this cycle I1121 14:06:06.412400 3798194 cpu.go:275] no items to output this cycle E1121 14:06:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:06:16.411985 3798194 memory.go:184] no items to output this cycle I1121 14:06:16.412159 3798194 cpu.go:275] no items to output this cycle I1121 14:06:17.480630 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:06:17.480638 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:06:26.412869 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:06:26.412905 3798194 memory.go:191] Add success. I1121 14:06:26.416988 3798194 cpu.go:282] Add success. I1121 14:06:26.441553 3798194 net.go:642] Add success. I1121 14:06:26.443058 3798194 net.go:778] primary dev: ETH0 I1121 14:06:26.443072 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:06:26.443086 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:06:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:06:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:06:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:06:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:06:36.412258 3798194 memory.go:184] no items to output this cycle I1121 14:06:36.412402 3798194 cpu.go:275] no items to output this cycle E1121 14:06:46.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:06:46.412250 3798194 memory.go:184] no items to output this cycle I1121 14:06:46.412407 3798194 cpu.go:275] no items to output this cycle E1121 14:06:56.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:06:56.413269 3798194 memory.go:191] Add success. W1121 14:06:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:06:56.413318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:06:56.413321 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:06:56.418018 3798194 cpu.go:282] Add success. I1121 14:06:56.441431 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:06:56.441931 3798194 net.go:642] Add success. I1121 14:06:56.443567 3798194 net.go:778] primary dev: ETH0 I1121 14:06:56.443585 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:06:56.443596 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:06:57.445290 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:06:57.445574 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:06:57.445587 3798194 disk_worker.go:723] disk space is not compliant W1121 14:06:57.445590 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:06:57.448153 3798194 disk_worker.go:509] system disk:vda1 I1121 14:06:57.448195 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:06:57.552245 3798194 disk_info.go:125] begin check local disk info of client I1121 14:06:57.555630 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:06:57.555639 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000322d40 0xc000322d80] I1121 14:06:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:06:59.450480 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:06:59.450558 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:06:59.450584 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:06:59.471098 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:07:06.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:07:06.412269 3798194 memory.go:184] no items to output this cycle I1121 14:07:06.412416 3798194 cpu.go:275] no items to output this cycle E1121 14:07:16.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:07:16.411988 3798194 memory.go:184] no items to output this cycle I1121 14:07:16.412145 3798194 cpu.go:275] no items to output this cycle E1121 14:07:26.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:07:26.411961 3798194 memory.go:191] Add success. I1121 14:07:26.412296 3798194 cpu.go:282] Add success. I1121 14:07:26.441920 3798194 net.go:642] Add success. I1121 14:07:26.446157 3798194 net.go:778] primary dev: ETH0 I1121 14:07:26.446172 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:07:26.446187 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:07:29.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:07:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:07:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:07:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:07:36.412256 3798194 memory.go:184] no items to output this cycle I1121 14:07:36.412416 3798194 cpu.go:275] no items to output this cycle E1121 14:07:46.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:07:46.413270 3798194 memory.go:184] no items to output this cycle I1121 14:07:46.413451 3798194 cpu.go:275] no items to output this cycle E1121 14:07:56.412250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:07:56.412289 3798194 memory.go:191] Add success. W1121 14:07:56.412324 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:07:56.412335 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:07:56.412339 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:07:56.416306 3798194 cpu.go:282] Add success. I1121 14:07:56.441814 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:07:56.442361 3798194 net.go:642] Add success. I1121 14:07:56.443812 3798194 net.go:778] primary dev: ETH0 I1121 14:07:56.443827 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:07:56.443840 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:07:57.452882 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:07:57.453046 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:07:57.453102 3798194 disk_worker.go:723] disk space is not compliant W1121 14:07:57.453105 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:07:57.463264 3798194 disk_worker.go:509] system disk:vda1 I1121 14:07:57.463305 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:07:57.555739 3798194 disk_info.go:125] begin check local disk info of client I1121 14:07:57.559262 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:07:57.559269 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0680 0xc0002e06c0] I1121 14:07:58.446391 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:07:59.450481 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:07:59.450572 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:07:59.450598 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:07:59.471137 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:08:06.413261 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:08:06.413290 3798194 memory.go:184] no items to output this cycle I1121 14:08:06.413448 3798194 cpu.go:275] no items to output this cycle E1121 14:08:16.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:08:16.413255 3798194 memory.go:184] no items to output this cycle I1121 14:08:16.413406 3798194 cpu.go:275] no items to output this cycle E1121 14:08:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:08:26.411952 3798194 memory.go:191] Add success. I1121 14:08:26.412282 3798194 cpu.go:282] Add success. I1121 14:08:26.441774 3798194 net.go:642] Add success. I1121 14:08:26.446294 3798194 net.go:778] primary dev: ETH0 I1121 14:08:26.446311 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:08:26.446322 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:08:29.450060 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:08:29.450149 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:08:29.450178 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:08:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:08:36.413254 3798194 memory.go:184] no items to output this cycle I1121 14:08:36.413396 3798194 cpu.go:275] no items to output this cycle E1121 14:08:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:08:46.411980 3798194 memory.go:184] no items to output this cycle I1121 14:08:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 14:08:56.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:08:56.413269 3798194 memory.go:191] Add success. W1121 14:08:56.413296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:08:56.413309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:08:56.413313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:08:56.417530 3798194 cpu.go:282] Add success. I1121 14:08:56.441933 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:08:56.442529 3798194 net.go:642] Add success. I1121 14:08:56.444032 3798194 net.go:778] primary dev: ETH0 I1121 14:08:56.444047 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:08:56.444061 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:08:57.449944 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:08:57.450101 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:08:57.450177 3798194 disk_worker.go:723] disk space is not compliant W1121 14:08:57.450181 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:08:57.458215 3798194 disk_worker.go:509] system disk:vda1 I1121 14:08:57.458258 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:08:57.559354 3798194 disk_info.go:125] begin check local disk info of client I1121 14:08:57.563169 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:08:57.563179 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004ac000 0xc0004ac040] I1121 14:08:58.446400 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:08:59.450505 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:08:59.450587 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:08:59.450615 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:08:59.471126 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:09:06.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:09:06.413261 3798194 memory.go:184] no items to output this cycle I1121 14:09:06.413404 3798194 cpu.go:275] no items to output this cycle E1121 14:09:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:09:16.411984 3798194 memory.go:184] no items to output this cycle I1121 14:09:16.412144 3798194 cpu.go:275] no items to output this cycle I1121 14:09:17.481626 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:09:17.481634 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:09:26.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:09:26.412893 3798194 memory.go:191] Add success. I1121 14:09:26.416975 3798194 cpu.go:282] Add success. I1121 14:09:26.441534 3798194 net.go:642] Add success. I1121 14:09:26.442946 3798194 net.go:778] primary dev: ETH0 I1121 14:09:26.442961 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:09:26.442974 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:09:29.449671 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:09:29.449748 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:09:29.449776 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:09:36.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:09:36.412251 3798194 memory.go:184] no items to output this cycle I1121 14:09:36.412391 3798194 cpu.go:275] no items to output this cycle E1121 14:09:46.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:09:46.413250 3798194 memory.go:184] no items to output this cycle I1121 14:09:46.413395 3798194 cpu.go:275] no items to output this cycle E1121 14:09:56.413471 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:09:56.413514 3798194 memory.go:191] Add success. W1121 14:09:56.413558 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:09:56.413574 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:09:56.413579 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:09:56.417736 3798194 cpu.go:282] Add success. I1121 14:09:56.441189 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:09:56.441671 3798194 net.go:642] Add success. I1121 14:09:56.443080 3798194 net.go:778] primary dev: ETH0 I1121 14:09:56.443097 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:09:56.443120 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:09:57.453876 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:09:57.454073 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:09:57.454084 3798194 disk_worker.go:723] disk space is not compliant W1121 14:09:57.454087 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:09:57.463001 3798194 disk_worker.go:509] system disk:vda1 I1121 14:09:57.463046 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:09:57.563272 3798194 disk_info.go:125] begin check local disk info of client I1121 14:09:57.566506 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:09:57.566514 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00039b3c0 0xc00039b400] I1121 14:09:58.445963 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:09:59.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:09:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:09:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:09:59.471648 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:10:06.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:10:06.413258 3798194 memory.go:184] no items to output this cycle I1121 14:10:06.413412 3798194 cpu.go:275] no items to output this cycle E1121 14:10:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:10:16.413258 3798194 memory.go:184] no items to output this cycle I1121 14:10:16.413410 3798194 cpu.go:275] no items to output this cycle E1121 14:10:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:10:26.413263 3798194 memory.go:191] Add success. I1121 14:10:26.417634 3798194 cpu.go:282] Add success. I1121 14:10:26.442177 3798194 net.go:642] Add success. I1121 14:10:26.443533 3798194 net.go:778] primary dev: ETH0 I1121 14:10:26.443549 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:10:26.443564 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:10:29.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:10:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:10:29.450501 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:10:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:10:36.411979 3798194 memory.go:184] no items to output this cycle I1121 14:10:36.412327 3798194 cpu.go:275] no items to output this cycle E1121 14:10:46.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:10:46.412254 3798194 memory.go:184] no items to output this cycle I1121 14:10:46.412405 3798194 cpu.go:275] no items to output this cycle E1121 14:10:56.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:10:56.413264 3798194 memory.go:191] Add success. W1121 14:10:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:10:56.413316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:10:56.413320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:10:56.417330 3798194 cpu.go:282] Add success. I1121 14:10:56.444296 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:10:56.444787 3798194 net.go:642] Add success. I1121 14:10:56.446457 3798194 net.go:778] primary dev: ETH0 I1121 14:10:56.446471 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:10:56.446483 3798194 net.go:659] Add eni metrics success, count: 7 E1121 14:10:57.450550 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 14:10:57.450564 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:10:57.450765 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:10:57.450775 3798194 disk_worker.go:723] disk space is not compliant W1121 14:10:57.450778 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:10:57.458787 3798194 disk_worker.go:509] system disk:vda1 I1121 14:10:57.458832 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:10:57.567243 3798194 disk_info.go:125] begin check local disk info of client I1121 14:10:57.570454 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:10:57.570462 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fad80 0xc0003fadc0] E1121 14:10:58.448143 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 14:10:58.448158 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 14:10:59.448779 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 14:10:59.449854 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:10:59.449909 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:10:59.449929 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:10:59.471473 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:11:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:11:06.412252 3798194 memory.go:184] no items to output this cycle I1121 14:11:06.412397 3798194 cpu.go:275] no items to output this cycle E1121 14:11:16.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:11:16.413257 3798194 memory.go:184] no items to output this cycle I1121 14:11:16.413400 3798194 cpu.go:275] no items to output this cycle E1121 14:11:26.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:11:26.412264 3798194 memory.go:191] Add success. I1121 14:11:26.416422 3798194 cpu.go:282] Add success. I1121 14:11:26.441962 3798194 net.go:642] Add success. I1121 14:11:26.443393 3798194 net.go:778] primary dev: ETH0 I1121 14:11:26.443408 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:11:26.443422 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:11:29.450030 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:11:29.450116 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:11:29.450148 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:11:36.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:11:36.413252 3798194 memory.go:184] no items to output this cycle I1121 14:11:36.413394 3798194 cpu.go:275] no items to output this cycle E1121 14:11:46.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:11:46.412880 3798194 memory.go:184] no items to output this cycle I1121 14:11:46.413022 3798194 cpu.go:275] no items to output this cycle E1121 14:11:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:11:56.412266 3798194 memory.go:191] Add success. W1121 14:11:56.412293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:11:56.412304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:11:56.412308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:11:56.416572 3798194 cpu.go:282] Add success. I1121 14:11:56.440937 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:11:56.441502 3798194 net.go:642] Add success. I1121 14:11:56.443060 3798194 net.go:778] primary dev: ETH0 I1121 14:11:56.443075 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:11:56.443088 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:11:57.448848 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:11:57.449076 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:11:57.449087 3798194 disk_worker.go:723] disk space is not compliant W1121 14:11:57.449091 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:11:57.451684 3798194 disk_worker.go:509] system disk:vda1 I1121 14:11:57.451723 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:11:57.570560 3798194 disk_info.go:125] begin check local disk info of client I1121 14:11:57.573868 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:11:57.573877 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004938c0 0xc000493900] I1121 14:11:58.446356 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:11:59.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:11:59.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:11:59.450510 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:11:59.471044 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:12:06.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:12:06.413261 3798194 memory.go:184] no items to output this cycle I1121 14:12:06.413404 3798194 cpu.go:275] no items to output this cycle E1121 14:12:16.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:12:16.413253 3798194 memory.go:184] no items to output this cycle I1121 14:12:16.413395 3798194 cpu.go:275] no items to output this cycle I1121 14:12:17.482632 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:12:17.482641 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:12:26.411319 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:12:26.411354 3798194 memory.go:191] Add success. I1121 14:12:26.411665 3798194 cpu.go:282] Add success. I1121 14:12:26.441243 3798194 net.go:642] Add success. I1121 14:12:26.445126 3798194 net.go:778] primary dev: ETH0 I1121 14:12:26.445142 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:12:26.445157 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:12:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:12:29.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:12:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:12:36.412815 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:12:36.412841 3798194 memory.go:184] no items to output this cycle I1121 14:12:36.412981 3798194 cpu.go:275] no items to output this cycle E1121 14:12:46.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:12:46.412255 3798194 memory.go:184] no items to output this cycle I1121 14:12:46.412398 3798194 cpu.go:275] no items to output this cycle E1121 14:12:56.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:12:56.412273 3798194 memory.go:191] Add success. W1121 14:12:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:12:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:12:56.412314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:12:56.416605 3798194 cpu.go:282] Add success. I1121 14:12:56.440924 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:12:56.441479 3798194 net.go:642] Add success. I1121 14:12:56.442899 3798194 net.go:778] primary dev: ETH0 I1121 14:12:56.442913 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:12:56.442924 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:12:57.445390 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:12:57.445591 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:12:57.445654 3798194 disk_worker.go:723] disk space is not compliant W1121 14:12:57.445658 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:12:57.447738 3798194 disk_worker.go:509] system disk:vda1 I1121 14:12:57.447779 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:12:57.573968 3798194 disk_info.go:125] begin check local disk info of client I1121 14:12:57.577220 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:12:57.577228 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003dab40 0xc0003dab80] I1121 14:12:58.445987 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:12:59.450496 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:12:59.450573 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:12:59.450597 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:12:59.471128 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:13:06.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:13:06.413251 3798194 memory.go:184] no items to output this cycle I1121 14:13:06.413396 3798194 cpu.go:275] no items to output this cycle E1121 14:13:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:13:16.411972 3798194 memory.go:184] no items to output this cycle I1121 14:13:16.412138 3798194 cpu.go:275] no items to output this cycle E1121 14:13:26.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:13:26.412271 3798194 memory.go:191] Add success. I1121 14:13:26.417015 3798194 cpu.go:282] Add success. I1121 14:13:26.441295 3798194 net.go:642] Add success. I1121 14:13:26.442925 3798194 net.go:778] primary dev: ETH0 I1121 14:13:26.442944 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:13:26.442958 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:13:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:13:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:13:29.450501 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:13:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:13:36.412252 3798194 memory.go:184] no items to output this cycle I1121 14:13:36.412397 3798194 cpu.go:275] no items to output this cycle E1121 14:13:46.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:13:46.413264 3798194 memory.go:184] no items to output this cycle I1121 14:13:46.413412 3798194 cpu.go:275] no items to output this cycle E1121 14:13:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:13:56.413263 3798194 memory.go:191] Add success. W1121 14:13:56.413291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:13:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:13:56.413306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:13:56.417735 3798194 cpu.go:282] Add success. I1121 14:13:56.441192 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:13:56.441687 3798194 net.go:642] Add success. I1121 14:13:56.443066 3798194 net.go:778] primary dev: ETH0 I1121 14:13:56.443081 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:13:56.443093 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:13:57.450694 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:13:57.450889 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:13:57.450901 3798194 disk_worker.go:723] disk space is not compliant W1121 14:13:57.450904 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:13:57.459265 3798194 disk_worker.go:509] system disk:vda1 I1121 14:13:57.459306 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:13:57.577327 3798194 disk_info.go:125] begin check local disk info of client I1121 14:13:57.580819 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:13:57.580829 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000308700 0xc000308740] I1121 14:13:58.446389 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:13:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:13:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:13:59.450506 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:13:59.471010 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:14:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:14:06.411981 3798194 memory.go:184] no items to output this cycle I1121 14:14:06.412142 3798194 cpu.go:275] no items to output this cycle E1121 14:14:16.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:14:16.413259 3798194 memory.go:184] no items to output this cycle I1121 14:14:16.413416 3798194 cpu.go:275] no items to output this cycle E1121 14:14:26.413177 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:14:26.413212 3798194 memory.go:191] Add success. I1121 14:14:26.417849 3798194 cpu.go:282] Add success. I1121 14:14:26.441288 3798194 net.go:642] Add success. I1121 14:14:26.442780 3798194 net.go:778] primary dev: ETH0 I1121 14:14:26.442795 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:14:26.442811 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:14:29.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:14:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:14:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:14:36.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:14:36.412886 3798194 memory.go:184] no items to output this cycle I1121 14:14:36.413033 3798194 cpu.go:275] no items to output this cycle E1121 14:14:46.411238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:14:46.411260 3798194 memory.go:184] no items to output this cycle I1121 14:14:46.411616 3798194 cpu.go:275] no items to output this cycle E1121 14:14:56.411932 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:14:56.411966 3798194 memory.go:191] Add success. W1121 14:14:56.412001 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:14:56.412013 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:14:56.412016 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:14:56.412366 3798194 cpu.go:282] Add success. I1121 14:14:56.441808 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:14:56.442327 3798194 net.go:642] Add success. I1121 14:14:56.446313 3798194 net.go:778] primary dev: ETH0 I1121 14:14:56.446329 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:14:56.446341 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:14:57.450495 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:14:57.450660 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:14:57.450714 3798194 disk_worker.go:723] disk space is not compliant W1121 14:14:57.450717 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:14:57.461236 3798194 disk_worker.go:509] system disk:vda1 I1121 14:14:57.461279 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:14:57.580921 3798194 disk_info.go:125] begin check local disk info of client I1121 14:14:57.584199 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:14:57.584207 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003da0c0 0xc0003da100] I1121 14:14:58.446384 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:14:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:14:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:14:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:14:59.471682 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:15:06.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:15:06.412254 3798194 memory.go:184] no items to output this cycle I1121 14:15:06.412402 3798194 cpu.go:275] no items to output this cycle E1121 14:15:16.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:15:16.412265 3798194 memory.go:184] no items to output this cycle I1121 14:15:16.412413 3798194 cpu.go:275] no items to output this cycle I1121 14:15:17.483264 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:15:17.483273 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:15:26.411336 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:15:26.411369 3798194 memory.go:191] Add success. I1121 14:15:26.411670 3798194 cpu.go:282] Add success. I1121 14:15:26.441306 3798194 net.go:642] Add success. I1121 14:15:26.444832 3798194 net.go:778] primary dev: ETH0 I1121 14:15:26.444847 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:15:26.444860 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:15:29.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:15:29.450129 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:15:29.450155 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:15:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:15:36.412256 3798194 memory.go:184] no items to output this cycle I1121 14:15:36.412393 3798194 cpu.go:275] no items to output this cycle E1121 14:15:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:15:46.413265 3798194 memory.go:184] no items to output this cycle I1121 14:15:46.413414 3798194 cpu.go:275] no items to output this cycle E1121 14:15:56.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:15:56.412265 3798194 memory.go:191] Add success. W1121 14:15:56.412291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:15:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:15:56.412305 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:15:56.416299 3798194 cpu.go:282] Add success. I1121 14:15:56.441445 3798194 event_worker.go:152] Polling the log file for events... I1121 14:15:56.444251 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:15:56.444787 3798194 net.go:642] Add success. I1121 14:15:56.446485 3798194 net.go:778] primary dev: ETH0 I1121 14:15:56.446501 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:15:56.446533 3798194 net.go:659] Add eni metrics success, count: 7 E1121 14:15:57.450585 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 14:15:57.450599 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:15:57.450752 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:15:57.450803 3798194 disk_worker.go:723] disk space is not compliant W1121 14:15:57.450806 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:15:57.458939 3798194 disk_worker.go:509] system disk:vda1 I1121 14:15:57.458980 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:15:57.584299 3798194 disk_info.go:125] begin check local disk info of client I1121 14:15:57.587522 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:15:57.587530 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071d00 0xc000071d40] E1121 14:15:58.447811 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 14:15:58.447825 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 14:15:59.448810 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 14:15:59.449887 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:15:59.449936 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:15:59.449958 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:15:59.471496 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:16:06.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:16:06.413258 3798194 memory.go:184] no items to output this cycle I1121 14:16:06.413402 3798194 cpu.go:275] no items to output this cycle E1121 14:16:16.412256 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:16:16.412288 3798194 memory.go:184] no items to output this cycle I1121 14:16:16.412438 3798194 cpu.go:275] no items to output this cycle E1121 14:16:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:16:26.413268 3798194 memory.go:191] Add success. I1121 14:16:26.417778 3798194 cpu.go:282] Add success. I1121 14:16:26.441244 3798194 net.go:642] Add success. I1121 14:16:26.443219 3798194 net.go:778] primary dev: ETH0 I1121 14:16:26.443236 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:16:26.443250 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:16:29.450132 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:16:29.450214 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:16:29.450239 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:16:36.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:16:36.412251 3798194 memory.go:184] no items to output this cycle I1121 14:16:36.412394 3798194 cpu.go:275] no items to output this cycle E1121 14:16:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:16:46.411977 3798194 memory.go:184] no items to output this cycle I1121 14:16:46.412130 3798194 cpu.go:275] no items to output this cycle E1121 14:16:56.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:16:56.413264 3798194 memory.go:191] Add success. W1121 14:16:56.413289 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:16:56.413300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:16:56.413304 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:16:56.417630 3798194 cpu.go:282] Add success. I1121 14:16:56.440969 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:16:56.441556 3798194 net.go:642] Add success. I1121 14:16:56.443075 3798194 net.go:778] primary dev: ETH0 I1121 14:16:56.443091 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:16:56.443103 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:16:57.447992 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:16:57.448214 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:16:57.448278 3798194 disk_worker.go:723] disk space is not compliant W1121 14:16:57.448282 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:16:57.452029 3798194 disk_worker.go:509] system disk:vda1 I1121 14:16:57.452068 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:16:57.587555 3798194 disk_info.go:125] begin check local disk info of client I1121 14:16:57.591189 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:16:57.591197 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fe040 0xc0003fe080] I1121 14:16:58.446384 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:16:59.450485 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:16:59.450566 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:16:59.450593 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:16:59.471105 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:17:06.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:17:06.413258 3798194 memory.go:184] no items to output this cycle I1121 14:17:06.413519 3798194 cpu.go:275] no items to output this cycle E1121 14:17:16.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:17:16.413251 3798194 memory.go:184] no items to output this cycle I1121 14:17:16.413399 3798194 cpu.go:275] no items to output this cycle E1121 14:17:26.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:17:26.413272 3798194 memory.go:191] Add success. I1121 14:17:26.417574 3798194 cpu.go:282] Add success. I1121 14:17:26.442090 3798194 net.go:642] Add success. I1121 14:17:26.443554 3798194 net.go:778] primary dev: ETH0 I1121 14:17:26.443571 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:17:26.443586 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:17:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:17:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:17:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:17:36.413037 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:17:36.413059 3798194 memory.go:184] no items to output this cycle I1121 14:17:36.413216 3798194 cpu.go:275] no items to output this cycle E1121 14:17:46.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:17:46.412248 3798194 memory.go:184] no items to output this cycle I1121 14:17:46.412394 3798194 cpu.go:275] no items to output this cycle E1121 14:17:56.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:17:56.413275 3798194 memory.go:191] Add success. W1121 14:17:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:17:56.413319 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:17:56.413322 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:17:56.417634 3798194 cpu.go:282] Add success. I1121 14:17:56.440961 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:17:56.441562 3798194 net.go:642] Add success. I1121 14:17:56.443148 3798194 net.go:778] primary dev: ETH0 I1121 14:17:56.443164 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:17:56.443178 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:17:57.450637 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:17:57.450815 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:17:57.450870 3798194 disk_worker.go:723] disk space is not compliant W1121 14:17:57.450874 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:17:57.459283 3798194 disk_worker.go:509] system disk:vda1 I1121 14:17:57.459326 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:17:57.591303 3798194 disk_info.go:125] begin check local disk info of client I1121 14:17:57.595087 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:17:57.595095 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000370e80 0xc000370ec0] I1121 14:17:58.446438 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:17:59.450533 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:17:59.450614 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:17:59.450643 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:17:59.471128 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:18:06.412785 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:18:06.412808 3798194 memory.go:184] no items to output this cycle I1121 14:18:06.412951 3798194 cpu.go:275] no items to output this cycle E1121 14:18:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:18:16.411983 3798194 memory.go:184] no items to output this cycle I1121 14:18:16.412138 3798194 cpu.go:275] no items to output this cycle I1121 14:18:17.483435 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:18:17.483443 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:18:26.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:18:26.413261 3798194 memory.go:191] Add success. I1121 14:18:26.417230 3798194 cpu.go:282] Add success. I1121 14:18:26.441876 3798194 net.go:642] Add success. I1121 14:18:26.443805 3798194 net.go:778] primary dev: ETH0 I1121 14:18:26.443823 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:18:26.443841 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:18:29.450227 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:18:29.450304 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:18:29.450330 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:18:36.412249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:18:36.412273 3798194 memory.go:184] no items to output this cycle I1121 14:18:36.412417 3798194 cpu.go:275] no items to output this cycle E1121 14:18:46.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:18:46.412257 3798194 memory.go:184] no items to output this cycle I1121 14:18:46.412400 3798194 cpu.go:275] no items to output this cycle E1121 14:18:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:18:56.411947 3798194 memory.go:191] Add success. W1121 14:18:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:18:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:18:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:18:56.412353 3798194 cpu.go:282] Add success. I1121 14:18:56.441833 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:18:56.442428 3798194 net.go:642] Add success. I1121 14:18:56.446593 3798194 net.go:778] primary dev: ETH0 I1121 14:18:56.446607 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:18:56.446620 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:18:57.450756 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:18:57.450970 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:18:57.450981 3798194 disk_worker.go:723] disk space is not compliant W1121 14:18:57.450985 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:18:57.459805 3798194 disk_worker.go:509] system disk:vda1 I1121 14:18:57.459847 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:18:57.596085 3798194 disk_info.go:125] begin check local disk info of client I1121 14:18:57.599723 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:18:57.599731 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f52c0 0xc0003f5300] I1121 14:18:58.446016 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:18:59.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:18:59.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:18:59.450150 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:18:59.471679 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:19:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:19:06.412255 3798194 memory.go:184] no items to output this cycle I1121 14:19:06.412400 3798194 cpu.go:275] no items to output this cycle E1121 14:19:16.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:19:16.412251 3798194 memory.go:184] no items to output this cycle I1121 14:19:16.412400 3798194 cpu.go:275] no items to output this cycle E1121 14:19:26.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:19:26.411952 3798194 memory.go:191] Add success. I1121 14:19:26.412298 3798194 cpu.go:282] Add success. I1121 14:19:26.441930 3798194 net.go:642] Add success. I1121 14:19:26.446180 3798194 net.go:778] primary dev: ETH0 I1121 14:19:26.446195 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:19:26.446209 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:19:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:19:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:19:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:19:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:19:36.413256 3798194 memory.go:184] no items to output this cycle I1121 14:19:36.413404 3798194 cpu.go:275] no items to output this cycle E1121 14:19:46.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:19:46.413254 3798194 memory.go:184] no items to output this cycle I1121 14:19:46.413398 3798194 cpu.go:275] no items to output this cycle E1121 14:19:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:19:56.411961 3798194 memory.go:191] Add success. W1121 14:19:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:19:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:19:56.412006 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:19:56.412385 3798194 cpu.go:282] Add success. I1121 14:19:56.441833 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:19:56.442337 3798194 net.go:642] Add success. I1121 14:19:56.446188 3798194 net.go:778] primary dev: ETH0 I1121 14:19:56.446203 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:19:56.446216 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:19:57.450628 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:19:57.450817 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:19:57.450828 3798194 disk_worker.go:723] disk space is not compliant W1121 14:19:57.450831 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:19:57.459172 3798194 disk_worker.go:509] system disk:vda1 I1121 14:19:57.459215 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:19:57.600196 3798194 disk_info.go:125] begin check local disk info of client I1121 14:19:57.603756 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:19:57.603764 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0e40 0xc0002e0e80] I1121 14:19:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:19:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:19:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:19:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:19:59.471712 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:20:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:20:06.412258 3798194 memory.go:184] no items to output this cycle I1121 14:20:06.412404 3798194 cpu.go:275] no items to output this cycle E1121 14:20:16.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:20:16.412249 3798194 memory.go:184] no items to output this cycle I1121 14:20:16.412398 3798194 cpu.go:275] no items to output this cycle E1121 14:20:26.413224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:20:26.413259 3798194 memory.go:191] Add success. I1121 14:20:26.417558 3798194 cpu.go:282] Add success. I1121 14:20:26.442104 3798194 net.go:642] Add success. I1121 14:20:26.443664 3798194 net.go:778] primary dev: ETH0 I1121 14:20:26.443678 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:20:26.443690 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:20:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:20:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:20:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:20:36.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:20:36.412873 3798194 memory.go:184] no items to output this cycle I1121 14:20:36.413017 3798194 cpu.go:275] no items to output this cycle E1121 14:20:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:20:46.411976 3798194 memory.go:184] no items to output this cycle I1121 14:20:46.412141 3798194 cpu.go:275] no items to output this cycle E1121 14:20:56.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:20:56.412264 3798194 memory.go:191] Add success. W1121 14:20:56.412291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:20:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:20:56.412306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:20:56.416884 3798194 cpu.go:282] Add success. I1121 14:20:56.444149 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:20:56.444647 3798194 net.go:642] Add success. I1121 14:20:56.446133 3798194 net.go:778] primary dev: ETH0 I1121 14:20:56.446150 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:20:56.446163 3798194 net.go:659] Add eni metrics success, count: 7 E1121 14:20:57.450570 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 14:20:57.450584 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:20:57.450856 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:20:57.450912 3798194 disk_worker.go:723] disk space is not compliant W1121 14:20:57.450916 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:20:57.459862 3798194 disk_worker.go:509] system disk:vda1 I1121 14:20:57.459901 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:20:57.603859 3798194 disk_info.go:125] begin check local disk info of client I1121 14:20:57.607424 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:20:57.607432 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004398c0 0xc000439900] E1121 14:20:58.448088 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 14:20:58.448102 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 14:20:59.448616 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 14:20:59.449690 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:20:59.449741 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:20:59.449760 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:20:59.471258 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:21:06.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:21:06.411981 3798194 memory.go:184] no items to output this cycle I1121 14:21:06.412137 3798194 cpu.go:275] no items to output this cycle E1121 14:21:16.412640 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:21:16.412664 3798194 memory.go:184] no items to output this cycle I1121 14:21:16.412814 3798194 cpu.go:275] no items to output this cycle I1121 14:21:17.483595 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:21:17.483603 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:21:26.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:21:26.413276 3798194 memory.go:191] Add success. I1121 14:21:26.417788 3798194 cpu.go:282] Add success. I1121 14:21:26.441279 3798194 net.go:642] Add success. I1121 14:21:26.442834 3798194 net.go:778] primary dev: ETH0 I1121 14:21:26.442849 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:21:26.442861 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:21:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:21:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:21:29.450498 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:21:36.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:21:36.412871 3798194 memory.go:184] no items to output this cycle I1121 14:21:36.413010 3798194 cpu.go:275] no items to output this cycle E1121 14:21:46.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:21:46.412262 3798194 memory.go:184] no items to output this cycle I1121 14:21:46.412408 3798194 cpu.go:275] no items to output this cycle E1121 14:21:56.412835 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:21:56.412868 3798194 memory.go:191] Add success. W1121 14:21:56.412895 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:21:56.412906 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:21:56.412910 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:21:56.417570 3798194 cpu.go:282] Add success. I1121 14:21:56.441942 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:21:56.442522 3798194 net.go:642] Add success. I1121 14:21:56.444070 3798194 net.go:778] primary dev: ETH0 I1121 14:21:56.444085 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:21:56.444097 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:21:57.450634 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:21:57.450792 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:21:57.450847 3798194 disk_worker.go:723] disk space is not compliant W1121 14:21:57.450850 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:21:57.459108 3798194 disk_worker.go:509] system disk:vda1 I1121 14:21:57.459166 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:21:57.607523 3798194 disk_info.go:125] begin check local disk info of client I1121 14:21:57.611289 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:21:57.611298 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000390c40 0xc000390c80] I1121 14:21:58.445997 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:21:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:21:59.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:21:59.450506 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:21:59.471008 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:22:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:22:06.412256 3798194 memory.go:184] no items to output this cycle I1121 14:22:06.412397 3798194 cpu.go:275] no items to output this cycle E1121 14:22:16.412871 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:22:16.412994 3798194 memory.go:184] no items to output this cycle I1121 14:22:16.413363 3798194 cpu.go:275] no items to output this cycle E1121 14:22:26.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:22:26.412267 3798194 memory.go:191] Add success. I1121 14:22:26.416666 3798194 cpu.go:282] Add success. I1121 14:22:26.442229 3798194 net.go:642] Add success. I1121 14:22:26.443770 3798194 net.go:778] primary dev: ETH0 I1121 14:22:26.443785 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:22:26.443798 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:22:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:22:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:22:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:22:36.412266 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:22:36.412294 3798194 memory.go:184] no items to output this cycle I1121 14:22:36.412447 3798194 cpu.go:275] no items to output this cycle E1121 14:22:46.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:22:46.412268 3798194 memory.go:184] no items to output this cycle I1121 14:22:46.412414 3798194 cpu.go:275] no items to output this cycle E1121 14:22:56.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:22:56.413270 3798194 memory.go:191] Add success. W1121 14:22:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:22:56.413318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:22:56.413322 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:22:56.417527 3798194 cpu.go:282] Add success. I1121 14:22:56.441946 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:22:56.442542 3798194 net.go:642] Add success. I1121 14:22:56.443957 3798194 net.go:778] primary dev: ETH0 I1121 14:22:56.443982 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:22:56.443994 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:22:57.450999 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:22:57.451233 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:22:57.451244 3798194 disk_worker.go:723] disk space is not compliant W1121 14:22:57.451247 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:22:57.462008 3798194 disk_worker.go:509] system disk:vda1 I1121 14:22:57.462052 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:22:57.611400 3798194 disk_info.go:125] begin check local disk info of client I1121 14:22:57.615143 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:22:57.615152 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002f9f00 0xc0002f9f40] I1121 14:22:58.446389 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:22:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:22:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:22:59.450506 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:22:59.471042 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:23:06.412259 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:23:06.412279 3798194 memory.go:184] no items to output this cycle I1121 14:23:06.412430 3798194 cpu.go:275] no items to output this cycle E1121 14:23:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:23:16.412261 3798194 memory.go:184] no items to output this cycle I1121 14:23:16.412408 3798194 cpu.go:275] no items to output this cycle E1121 14:23:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:23:26.411959 3798194 memory.go:191] Add success. I1121 14:23:26.412331 3798194 cpu.go:282] Add success. I1121 14:23:26.442036 3798194 net.go:642] Add success. I1121 14:23:26.445971 3798194 net.go:778] primary dev: ETH0 I1121 14:23:26.445986 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:23:26.446001 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:23:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:23:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:23:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:23:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:23:36.412256 3798194 memory.go:184] no items to output this cycle I1121 14:23:36.412400 3798194 cpu.go:275] no items to output this cycle E1121 14:23:46.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:23:46.411969 3798194 memory.go:184] no items to output this cycle I1121 14:23:46.412141 3798194 cpu.go:275] no items to output this cycle E1121 14:23:56.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:23:56.412891 3798194 memory.go:191] Add success. W1121 14:23:56.412925 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:23:56.412936 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:23:56.412940 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:23:56.417793 3798194 cpu.go:282] Add success. I1121 14:23:56.441192 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:23:56.441704 3798194 net.go:642] Add success. I1121 14:23:56.443436 3798194 net.go:778] primary dev: ETH0 I1121 14:23:56.443450 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:23:56.443462 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:23:57.451003 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:23:57.451244 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:23:57.451255 3798194 disk_worker.go:723] disk space is not compliant W1121 14:23:57.451258 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:23:57.460054 3798194 disk_worker.go:509] system disk:vda1 I1121 14:23:57.460095 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:23:57.615250 3798194 disk_info.go:125] begin check local disk info of client I1121 14:23:57.618948 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:23:57.618956 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e1840 0xc0002e1880] I1121 14:23:58.446132 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:23:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:23:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:23:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:23:59.471696 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:24:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:24:06.411984 3798194 memory.go:184] no items to output this cycle I1121 14:24:06.412140 3798194 cpu.go:275] no items to output this cycle E1121 14:24:16.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:24:16.411991 3798194 memory.go:184] no items to output this cycle I1121 14:24:16.412153 3798194 cpu.go:275] no items to output this cycle I1121 14:24:17.484063 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:24:17.484072 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:24:26.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:24:26.412270 3798194 memory.go:191] Add success. I1121 14:24:26.417032 3798194 cpu.go:282] Add success. I1121 14:24:26.441574 3798194 net.go:642] Add success. I1121 14:24:26.444800 3798194 net.go:778] primary dev: ETH0 I1121 14:24:26.444815 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:24:26.444827 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:24:29.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:24:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:24:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:24:36.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:24:36.413258 3798194 memory.go:184] no items to output this cycle I1121 14:24:36.413399 3798194 cpu.go:275] no items to output this cycle E1121 14:24:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:24:46.411974 3798194 memory.go:184] no items to output this cycle I1121 14:24:46.412131 3798194 cpu.go:275] no items to output this cycle E1121 14:24:56.411249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:24:56.411281 3798194 memory.go:191] Add success. W1121 14:24:56.411316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:24:56.411327 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:24:56.411331 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:24:56.416238 3798194 cpu.go:282] Add success. I1121 14:24:56.441611 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:24:56.442141 3798194 net.go:642] Add success. I1121 14:24:56.443766 3798194 net.go:778] primary dev: ETH0 I1121 14:24:56.443780 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:24:56.443793 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:24:57.451072 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:24:57.451251 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:24:57.451308 3798194 disk_worker.go:723] disk space is not compliant W1121 14:24:57.451311 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:24:57.460140 3798194 disk_worker.go:509] system disk:vda1 I1121 14:24:57.460183 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:24:57.619062 3798194 disk_info.go:125] begin check local disk info of client I1121 14:24:57.622834 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:24:57.622843 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004540c0 0xc000454100] I1121 14:24:58.446384 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:24:59.450386 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:24:59.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:24:59.450496 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:24:59.471032 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:25:06.411983 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:25:06.412005 3798194 memory.go:184] no items to output this cycle I1121 14:25:06.412161 3798194 cpu.go:275] no items to output this cycle E1121 14:25:16.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:25:16.412268 3798194 memory.go:184] no items to output this cycle I1121 14:25:16.412410 3798194 cpu.go:275] no items to output this cycle E1121 14:25:26.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:25:26.411950 3798194 memory.go:191] Add success. I1121 14:25:26.412310 3798194 cpu.go:282] Add success. I1121 14:25:26.441924 3798194 net.go:642] Add success. I1121 14:25:26.446130 3798194 net.go:778] primary dev: ETH0 I1121 14:25:26.446146 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:25:26.446159 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:25:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:25:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:25:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:25:36.412756 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:25:36.412782 3798194 memory.go:184] no items to output this cycle I1121 14:25:36.412931 3798194 cpu.go:275] no items to output this cycle E1121 14:25:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:25:46.411976 3798194 memory.go:184] no items to output this cycle I1121 14:25:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 14:25:56.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:25:56.413260 3798194 memory.go:191] Add success. W1121 14:25:56.413294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:25:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:25:56.413309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:25:56.417643 3798194 cpu.go:282] Add success. I1121 14:25:56.443602 3798194 event_worker.go:152] Polling the log file for events... I1121 14:25:56.443819 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:25:56.444326 3798194 net.go:642] Add success. I1121 14:25:56.445859 3798194 net.go:778] primary dev: ETH0 I1121 14:25:56.445873 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:25:56.445885 3798194 net.go:659] Add eni metrics success, count: 7 E1121 14:25:57.450967 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 14:25:57.450982 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:25:57.451194 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:25:57.451204 3798194 disk_worker.go:723] disk space is not compliant W1121 14:25:57.451217 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:25:57.459912 3798194 disk_worker.go:509] system disk:vda1 I1121 14:25:57.459955 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:25:57.623132 3798194 disk_info.go:125] begin check local disk info of client I1121 14:25:57.626671 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:25:57.626678 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00039f740 0xc00039f780] E1121 14:25:58.448499 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 14:25:58.448515 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 14:25:59.448984 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 14:25:59.450061 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:25:59.450130 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:25:59.450150 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:25:59.471687 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:26:06.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:26:06.411989 3798194 memory.go:184] no items to output this cycle I1121 14:26:06.412147 3798194 cpu.go:275] no items to output this cycle E1121 14:26:16.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:26:16.413267 3798194 memory.go:184] no items to output this cycle I1121 14:26:16.413411 3798194 cpu.go:275] no items to output this cycle E1121 14:26:26.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:26:26.411943 3798194 memory.go:191] Add success. I1121 14:26:26.412283 3798194 cpu.go:282] Add success. I1121 14:26:26.441765 3798194 net.go:642] Add success. I1121 14:26:26.453045 3798194 net.go:778] primary dev: ETH0 I1121 14:26:26.453060 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:26:26.453227 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:26:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:26:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:26:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:26:36.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:26:36.413256 3798194 memory.go:184] no items to output this cycle I1121 14:26:36.413400 3798194 cpu.go:275] no items to output this cycle E1121 14:26:46.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:26:46.412258 3798194 memory.go:184] no items to output this cycle I1121 14:26:46.412405 3798194 cpu.go:275] no items to output this cycle E1121 14:26:56.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:26:56.412263 3798194 memory.go:191] Add success. W1121 14:26:56.412296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:26:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:26:56.412312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:26:56.417151 3798194 cpu.go:282] Add success. I1121 14:26:56.441444 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:26:56.441941 3798194 net.go:642] Add success. I1121 14:26:56.443632 3798194 net.go:778] primary dev: ETH0 I1121 14:26:56.443647 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:26:56.443660 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:26:57.450997 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:26:57.451180 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:26:57.451237 3798194 disk_worker.go:723] disk space is not compliant W1121 14:26:57.451241 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:26:57.460008 3798194 disk_worker.go:509] system disk:vda1 I1121 14:26:57.460050 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:26:57.626780 3798194 disk_info.go:125] begin check local disk info of client I1121 14:26:57.630583 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:26:57.630593 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003db600 0xc0003db640] I1121 14:26:58.446381 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:26:59.450485 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:26:59.450567 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:26:59.450592 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:26:59.471126 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:27:06.411971 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:27:06.411995 3798194 memory.go:184] no items to output this cycle I1121 14:27:06.412158 3798194 cpu.go:275] no items to output this cycle E1121 14:27:16.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:27:16.411990 3798194 memory.go:184] no items to output this cycle I1121 14:27:16.412155 3798194 cpu.go:275] no items to output this cycle I1121 14:27:17.484220 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:27:17.484228 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:27:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:27:26.412266 3798194 memory.go:191] Add success. I1121 14:27:26.416466 3798194 cpu.go:282] Add success. I1121 14:27:26.442010 3798194 net.go:642] Add success. I1121 14:27:26.443439 3798194 net.go:778] primary dev: ETH0 I1121 14:27:26.443453 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:27:26.443466 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:27:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:27:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:27:29.450495 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:27:36.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:27:36.412878 3798194 memory.go:184] no items to output this cycle I1121 14:27:36.413020 3798194 cpu.go:275] no items to output this cycle E1121 14:27:46.412066 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:27:46.412088 3798194 memory.go:184] no items to output this cycle I1121 14:27:46.412252 3798194 cpu.go:275] no items to output this cycle E1121 14:27:56.412728 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:27:56.412763 3798194 memory.go:191] Add success. W1121 14:27:56.412798 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:27:56.412810 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:27:56.412813 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:27:56.417441 3798194 cpu.go:282] Add success. I1121 14:27:56.441864 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:27:56.442579 3798194 net.go:642] Add success. I1121 14:27:56.444265 3798194 net.go:778] primary dev: ETH0 I1121 14:27:56.444280 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:27:56.444299 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:27:57.451801 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:27:57.452022 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:27:57.452032 3798194 disk_worker.go:723] disk space is not compliant W1121 14:27:57.452036 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:27:57.460428 3798194 disk_worker.go:509] system disk:vda1 I1121 14:27:57.460470 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:27:57.630691 3798194 disk_info.go:125] begin check local disk info of client I1121 14:27:57.634322 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:27:57.634330 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004dd080 0xc0004dd0c0] I1121 14:27:58.445993 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:27:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:27:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:27:59.450504 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:27:59.471006 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:28:06.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:28:06.412879 3798194 memory.go:184] no items to output this cycle I1121 14:28:06.413025 3798194 cpu.go:275] no items to output this cycle E1121 14:28:16.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:28:16.413251 3798194 memory.go:184] no items to output this cycle I1121 14:28:16.413395 3798194 cpu.go:275] no items to output this cycle E1121 14:28:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:28:26.413266 3798194 memory.go:191] Add success. I1121 14:28:26.417625 3798194 cpu.go:282] Add success. I1121 14:28:26.442154 3798194 net.go:642] Add success. I1121 14:28:26.443860 3798194 net.go:778] primary dev: ETH0 I1121 14:28:26.443874 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:28:26.443889 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:28:29.449993 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:28:29.450080 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:28:29.450106 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:28:36.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:28:36.412888 3798194 memory.go:184] no items to output this cycle I1121 14:28:36.413031 3798194 cpu.go:275] no items to output this cycle E1121 14:28:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:28:46.412261 3798194 memory.go:184] no items to output this cycle I1121 14:28:46.412406 3798194 cpu.go:275] no items to output this cycle E1121 14:28:56.417698 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:28:56.417756 3798194 memory.go:191] Add success. W1121 14:28:56.417801 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:28:56.417817 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:28:56.417822 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:28:56.424087 3798194 cpu.go:282] Add success. I1121 14:28:56.441981 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:28:56.442823 3798194 net.go:642] Add success. I1121 14:28:56.444455 3798194 net.go:778] primary dev: ETH0 I1121 14:28:56.444477 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:28:56.444494 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:28:57.466234 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:28:57.466615 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:28:57.466632 3798194 disk_worker.go:723] disk space is not compliant W1121 14:28:57.466637 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:28:57.470242 3798194 disk_worker.go:509] system disk:vda1 I1121 14:28:57.470300 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:28:57.635219 3798194 disk_info.go:125] begin check local disk info of client I1121 14:28:57.640898 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:28:57.640909 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ebf00 0xc0003ebf40] I1121 14:28:58.446598 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:28:59.450421 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:28:59.450532 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:28:59.450572 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:28:59.471219 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:29:06.414254 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:29:06.414277 3798194 memory.go:184] no items to output this cycle I1121 14:29:06.414423 3798194 cpu.go:275] no items to output this cycle E1121 14:29:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:29:16.411977 3798194 memory.go:184] no items to output this cycle I1121 14:29:16.412333 3798194 cpu.go:275] no items to output this cycle E1121 14:29:26.411910 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:29:26.411944 3798194 memory.go:191] Add success. I1121 14:29:26.412327 3798194 cpu.go:282] Add success. I1121 14:29:26.441971 3798194 net.go:642] Add success. I1121 14:29:26.445914 3798194 net.go:778] primary dev: ETH0 I1121 14:29:26.445930 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:29:26.445944 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:29:29.449938 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:29:29.450024 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:29:29.450053 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:29:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:29:36.413258 3798194 memory.go:184] no items to output this cycle I1121 14:29:36.413399 3798194 cpu.go:275] no items to output this cycle E1121 14:29:46.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:29:46.412257 3798194 memory.go:184] no items to output this cycle I1121 14:29:46.412398 3798194 cpu.go:275] no items to output this cycle E1121 14:29:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:29:56.411956 3798194 memory.go:191] Add success. W1121 14:29:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:29:56.412012 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:29:56.412016 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:29:56.412365 3798194 cpu.go:282] Add success. I1121 14:29:56.441825 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:29:56.442422 3798194 net.go:642] Add success. I1121 14:29:56.446764 3798194 net.go:778] primary dev: ETH0 I1121 14:29:56.446778 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:29:56.446791 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:29:57.450599 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:29:57.450758 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:29:57.450810 3798194 disk_worker.go:723] disk space is not compliant W1121 14:29:57.450813 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:29:57.459194 3798194 disk_worker.go:509] system disk:vda1 I1121 14:29:57.459238 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:29:57.641147 3798194 disk_info.go:125] begin check local disk info of client I1121 14:29:57.644786 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:29:57.644794 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030db80 0xc00030dbc0] I1121 14:29:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:29:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:29:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:29:59.450133 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:29:59.471673 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:30:06.412100 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:30:06.412140 3798194 memory.go:184] no items to output this cycle I1121 14:30:06.412282 3798194 cpu.go:275] no items to output this cycle E1121 14:30:16.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:30:16.411970 3798194 memory.go:184] no items to output this cycle I1121 14:30:16.412122 3798194 cpu.go:275] no items to output this cycle I1121 14:30:17.485269 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:30:17.485278 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:30:26.411237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:30:26.411271 3798194 memory.go:191] Add success. I1121 14:30:26.411594 3798194 cpu.go:282] Add success. I1121 14:30:26.442200 3798194 net.go:642] Add success. I1121 14:30:26.446319 3798194 net.go:778] primary dev: ETH0 I1121 14:30:26.446334 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:30:26.446348 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:30:29.450412 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:30:29.450491 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:30:29.450519 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:30:36.413212 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:30:36.413235 3798194 memory.go:184] no items to output this cycle I1121 14:30:36.413377 3798194 cpu.go:275] no items to output this cycle E1121 14:30:46.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:30:46.412259 3798194 memory.go:184] no items to output this cycle I1121 14:30:46.412404 3798194 cpu.go:275] no items to output this cycle E1121 14:30:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:30:56.411953 3798194 memory.go:191] Add success. W1121 14:30:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:30:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:30:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:30:56.412365 3798194 cpu.go:282] Add success. I1121 14:30:56.445287 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:30:56.445865 3798194 net.go:642] Add success. I1121 14:30:56.454773 3798194 net.go:778] primary dev: ETH0 I1121 14:30:56.454788 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:30:56.454801 3798194 net.go:659] Add eni metrics success, count: 7 E1121 14:30:57.450980 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 14:30:57.450994 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:30:57.451210 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:30:57.451221 3798194 disk_worker.go:723] disk space is not compliant W1121 14:30:57.451224 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:30:57.460059 3798194 disk_worker.go:509] system disk:vda1 I1121 14:30:57.460100 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:30:57.645142 3798194 disk_info.go:125] begin check local disk info of client I1121 14:30:57.648839 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:30:57.648848 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e1580 0xc0002e15c0] E1121 14:30:58.448457 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 14:30:58.448472 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:30:59.450482 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:30:59.450537 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:30:59.450556 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:30:59.450680 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 14:30:59.471101 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:31:06.412762 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:31:06.412785 3798194 memory.go:184] no items to output this cycle I1121 14:31:06.412945 3798194 cpu.go:275] no items to output this cycle E1121 14:31:16.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:31:16.412261 3798194 memory.go:184] no items to output this cycle I1121 14:31:16.412407 3798194 cpu.go:275] no items to output this cycle E1121 14:31:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:31:26.411944 3798194 memory.go:191] Add success. I1121 14:31:26.412300 3798194 cpu.go:282] Add success. I1121 14:31:26.441925 3798194 net.go:642] Add success. I1121 14:31:26.445896 3798194 net.go:778] primary dev: ETH0 I1121 14:31:26.445912 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:31:26.445925 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:31:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:31:29.450487 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:31:29.450516 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:31:36.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:31:36.413255 3798194 memory.go:184] no items to output this cycle I1121 14:31:36.413396 3798194 cpu.go:275] no items to output this cycle E1121 14:31:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:31:46.411980 3798194 memory.go:184] no items to output this cycle I1121 14:31:46.412138 3798194 cpu.go:275] no items to output this cycle E1121 14:31:56.412299 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:31:56.412333 3798194 memory.go:191] Add success. W1121 14:31:56.412360 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:31:56.412371 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:31:56.412374 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:31:56.416520 3798194 cpu.go:282] Add success. I1121 14:31:56.441930 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:31:56.442507 3798194 net.go:642] Add success. I1121 14:31:56.444078 3798194 net.go:778] primary dev: ETH0 I1121 14:31:56.444092 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:31:56.444107 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:31:57.451066 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:31:57.451274 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:31:57.451286 3798194 disk_worker.go:723] disk space is not compliant W1121 14:31:57.451289 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:31:57.460019 3798194 disk_worker.go:509] system disk:vda1 I1121 14:31:57.460059 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:31:57.648948 3798194 disk_info.go:125] begin check local disk info of client I1121 14:31:57.652699 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:31:57.652707 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047ca00 0xc00047ca40] I1121 14:31:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:31:59.450105 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:31:59.450194 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:31:59.450219 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:31:59.470696 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:32:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:32:06.411983 3798194 memory.go:184] no items to output this cycle I1121 14:32:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 14:32:16.412246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:32:16.412269 3798194 memory.go:184] no items to output this cycle I1121 14:32:16.412417 3798194 cpu.go:275] no items to output this cycle E1121 14:32:26.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:32:26.412885 3798194 memory.go:191] Add success. I1121 14:32:26.417193 3798194 cpu.go:282] Add success. I1121 14:32:26.441664 3798194 net.go:642] Add success. I1121 14:32:26.443222 3798194 net.go:778] primary dev: ETH0 I1121 14:32:26.443236 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:32:26.443248 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:32:29.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:32:29.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:32:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:32:36.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:32:36.413261 3798194 memory.go:184] no items to output this cycle I1121 14:32:36.413406 3798194 cpu.go:275] no items to output this cycle E1121 14:32:46.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:32:46.412262 3798194 memory.go:184] no items to output this cycle I1121 14:32:46.412411 3798194 cpu.go:275] no items to output this cycle E1121 14:32:56.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:32:56.412278 3798194 memory.go:191] Add success. W1121 14:32:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:32:56.412324 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:32:56.412327 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:32:56.416678 3798194 cpu.go:282] Add success. I1121 14:32:56.440949 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:32:56.441530 3798194 net.go:642] Add success. I1121 14:32:56.443047 3798194 net.go:778] primary dev: ETH0 I1121 14:32:56.443062 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:32:56.443074 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:32:57.450948 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:32:57.451106 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:32:57.451184 3798194 disk_worker.go:723] disk space is not compliant W1121 14:32:57.451188 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:32:57.459919 3798194 disk_worker.go:509] system disk:vda1 I1121 14:32:57.459960 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:32:57.653165 3798194 disk_info.go:125] begin check local disk info of client I1121 14:32:57.656821 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:32:57.656828 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e6140 0xc0003e6180] I1121 14:32:58.446277 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:32:59.450413 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:32:59.450491 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:32:59.450517 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:32:59.471034 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:33:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:33:06.412257 3798194 memory.go:184] no items to output this cycle I1121 14:33:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 14:33:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:33:16.411975 3798194 memory.go:184] no items to output this cycle I1121 14:33:16.412132 3798194 cpu.go:275] no items to output this cycle I1121 14:33:17.486265 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:33:17.486273 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:33:26.411201 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:33:26.411236 3798194 memory.go:191] Add success. I1121 14:33:26.411583 3798194 cpu.go:282] Add success. I1121 14:33:26.442342 3798194 net.go:642] Add success. I1121 14:33:26.446230 3798194 net.go:778] primary dev: ETH0 I1121 14:33:26.446246 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:33:26.446265 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:33:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:33:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:33:29.450144 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:33:36.412803 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:33:36.412827 3798194 memory.go:184] no items to output this cycle I1121 14:33:36.412986 3798194 cpu.go:275] no items to output this cycle E1121 14:33:46.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:33:46.411985 3798194 memory.go:184] no items to output this cycle I1121 14:33:46.412140 3798194 cpu.go:275] no items to output this cycle E1121 14:33:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:33:56.411952 3798194 memory.go:191] Add success. W1121 14:33:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:33:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:33:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:33:56.412335 3798194 cpu.go:282] Add success. I1121 14:33:56.441816 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:33:56.442368 3798194 net.go:642] Add success. I1121 14:33:56.446306 3798194 net.go:778] primary dev: ETH0 I1121 14:33:56.446322 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:33:56.446334 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:33:57.450693 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:33:57.450886 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:33:57.450898 3798194 disk_worker.go:723] disk space is not compliant W1121 14:33:57.450902 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:33:57.459242 3798194 disk_worker.go:509] system disk:vda1 I1121 14:33:57.459287 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:33:57.656922 3798194 disk_info.go:125] begin check local disk info of client I1121 14:33:57.660735 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:33:57.660745 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000488580 0xc0004885c0] I1121 14:33:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:33:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:33:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:33:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:33:59.471665 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:34:06.413168 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:34:06.413192 3798194 memory.go:184] no items to output this cycle I1121 14:34:06.413340 3798194 cpu.go:275] no items to output this cycle E1121 14:34:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:34:16.411973 3798194 memory.go:184] no items to output this cycle I1121 14:34:16.412126 3798194 cpu.go:275] no items to output this cycle E1121 14:34:26.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:34:26.412886 3798194 memory.go:191] Add success. I1121 14:34:26.417309 3798194 cpu.go:282] Add success. I1121 14:34:26.441721 3798194 net.go:642] Add success. I1121 14:34:26.443242 3798194 net.go:778] primary dev: ETH0 I1121 14:34:26.443260 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:34:26.443274 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:34:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:34:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:34:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:34:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:34:36.411976 3798194 memory.go:184] no items to output this cycle I1121 14:34:36.412127 3798194 cpu.go:275] no items to output this cycle E1121 14:34:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:34:46.411981 3798194 memory.go:184] no items to output this cycle I1121 14:34:46.412144 3798194 cpu.go:275] no items to output this cycle E1121 14:34:56.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:34:56.412894 3798194 memory.go:191] Add success. W1121 14:34:56.412921 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:34:56.412933 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:34:56.412936 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:34:56.417227 3798194 cpu.go:282] Add success. I1121 14:34:56.441612 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:34:56.442528 3798194 net.go:642] Add success. I1121 14:34:56.444189 3798194 net.go:778] primary dev: ETH0 I1121 14:34:56.444204 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:34:56.444312 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:34:57.451174 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:34:57.451335 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:34:57.451386 3798194 disk_worker.go:723] disk space is not compliant W1121 14:34:57.451389 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:34:57.459763 3798194 disk_worker.go:509] system disk:vda1 I1121 14:34:57.459804 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:34:57.660843 3798194 disk_info.go:125] begin check local disk info of client I1121 14:34:57.664641 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:34:57.664650 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004cf8c0 0xc0004cf900] I1121 14:34:58.446643 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:34:59.449666 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:34:59.449747 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:34:59.449772 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:34:59.471318 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:35:06.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:35:06.412886 3798194 memory.go:184] no items to output this cycle I1121 14:35:06.413027 3798194 cpu.go:275] no items to output this cycle E1121 14:35:16.412688 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:35:16.412711 3798194 memory.go:184] no items to output this cycle I1121 14:35:16.412862 3798194 cpu.go:275] no items to output this cycle E1121 14:35:26.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:35:26.412889 3798194 memory.go:191] Add success. I1121 14:35:26.417476 3798194 cpu.go:282] Add success. I1121 14:35:26.442030 3798194 net.go:642] Add success. I1121 14:35:26.443559 3798194 net.go:778] primary dev: ETH0 I1121 14:35:26.443574 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:35:26.443587 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:35:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:35:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:35:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:35:36.413174 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:35:36.413195 3798194 memory.go:184] no items to output this cycle I1121 14:35:36.413339 3798194 cpu.go:275] no items to output this cycle E1121 14:35:46.412773 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:35:46.412796 3798194 memory.go:184] no items to output this cycle I1121 14:35:46.412938 3798194 cpu.go:275] no items to output this cycle E1121 14:35:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:35:56.412266 3798194 memory.go:191] Add success. W1121 14:35:56.412292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:35:56.412304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:35:56.412307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:35:56.416443 3798194 cpu.go:282] Add success. I1121 14:35:56.441618 3798194 event_worker.go:152] Polling the log file for events... I1121 14:35:56.444852 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:35:56.445368 3798194 net.go:642] Add success. I1121 14:35:56.446801 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795241156, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 14:35:56.446912 3798194 net.go:778] primary dev: ETH0 I1121 14:35:56.446926 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:35:56.446938 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:35:56.454870 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 14:35:57.446468 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 14:35:57.446482 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:35:57.446734 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:35:57.446745 3798194 disk_worker.go:723] disk space is not compliant W1121 14:35:57.446748 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:35:57.449540 3798194 disk_worker.go:509] system disk:vda1 I1121 14:35:57.449583 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:35:57.664742 3798194 disk_info.go:125] begin check local disk info of client I1121 14:35:57.668590 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:35:57.668599 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000416a80 0xc000416ac0] E1121 14:35:58.448181 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 14:35:58.448196 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 14:35:59.448645 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 14:35:59.449723 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:35:59.449774 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:35:59.449800 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:35:59.471332 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:36:06.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:36:06.411985 3798194 memory.go:184] no items to output this cycle I1121 14:36:06.412153 3798194 cpu.go:275] no items to output this cycle E1121 14:36:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:36:16.411986 3798194 memory.go:184] no items to output this cycle I1121 14:36:16.412141 3798194 cpu.go:275] no items to output this cycle I1121 14:36:17.487279 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:36:17.487288 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:36:26.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:36:26.412261 3798194 memory.go:191] Add success. I1121 14:36:26.417006 3798194 cpu.go:282] Add success. I1121 14:36:26.441512 3798194 net.go:642] Add success. I1121 14:36:26.442940 3798194 net.go:778] primary dev: ETH0 I1121 14:36:26.442954 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:36:26.442967 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:36:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:36:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:36:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:36:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:36:36.411976 3798194 memory.go:184] no items to output this cycle I1121 14:36:36.412134 3798194 cpu.go:275] no items to output this cycle E1121 14:36:46.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:36:46.412872 3798194 memory.go:184] no items to output this cycle I1121 14:36:46.413016 3798194 cpu.go:275] no items to output this cycle E1121 14:36:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:36:56.411953 3798194 memory.go:191] Add success. W1121 14:36:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:36:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:36:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:36:56.412347 3798194 cpu.go:282] Add success. I1121 14:36:56.441803 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:36:56.442364 3798194 net.go:642] Add success. I1121 14:36:56.446461 3798194 net.go:778] primary dev: ETH0 I1121 14:36:56.446477 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:36:56.446490 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:36:57.450859 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:36:57.451041 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:36:57.451096 3798194 disk_worker.go:723] disk space is not compliant W1121 14:36:57.451100 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:36:57.460031 3798194 disk_worker.go:509] system disk:vda1 I1121 14:36:57.460076 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:36:57.668698 3798194 disk_info.go:125] begin check local disk info of client I1121 14:36:57.672499 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:36:57.672508 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037e280 0xc00037e2c0] I1121 14:36:58.446375 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:36:59.450473 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:36:59.450558 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:36:59.450582 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:36:59.471118 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:37:06.413268 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:37:06.413290 3798194 memory.go:184] no items to output this cycle I1121 14:37:06.413453 3798194 cpu.go:275] no items to output this cycle E1121 14:37:16.412921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:37:16.412943 3798194 memory.go:184] no items to output this cycle I1121 14:37:16.413095 3798194 cpu.go:275] no items to output this cycle E1121 14:37:26.412223 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:37:26.412265 3798194 memory.go:191] Add success. I1121 14:37:26.416279 3798194 cpu.go:282] Add success. I1121 14:37:26.441657 3798194 net.go:642] Add success. I1121 14:37:26.443171 3798194 net.go:778] primary dev: ETH0 I1121 14:37:26.443185 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:37:26.443197 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:37:29.450448 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:37:29.450525 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:37:29.450551 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:37:36.413368 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:37:36.413403 3798194 memory.go:184] no items to output this cycle I1121 14:37:36.413551 3798194 cpu.go:275] no items to output this cycle E1121 14:37:46.412096 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:37:46.412201 3798194 memory.go:184] no items to output this cycle I1121 14:37:46.412631 3798194 cpu.go:275] no items to output this cycle E1121 14:37:56.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:37:56.411958 3798194 memory.go:191] Add success. W1121 14:37:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:37:56.412005 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:37:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:37:56.412396 3798194 cpu.go:282] Add success. I1121 14:37:56.441848 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:37:56.442363 3798194 net.go:642] Add success. I1121 14:37:56.446823 3798194 net.go:778] primary dev: ETH0 I1121 14:37:56.446839 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:37:56.446851 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:37:57.452265 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:37:57.452470 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:37:57.452482 3798194 disk_worker.go:723] disk space is not compliant W1121 14:37:57.452485 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:37:57.463502 3798194 disk_worker.go:509] system disk:vda1 I1121 14:37:57.463545 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:37:57.672608 3798194 disk_info.go:125] begin check local disk info of client I1121 14:37:57.676448 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:37:57.676457 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000472400 0xc000472440] I1121 14:37:58.446011 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:37:59.450414 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:37:59.450509 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:37:59.450542 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:37:59.471083 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:38:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:38:06.412261 3798194 memory.go:184] no items to output this cycle I1121 14:38:06.412403 3798194 cpu.go:275] no items to output this cycle E1121 14:38:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:38:16.411973 3798194 memory.go:184] no items to output this cycle I1121 14:38:16.412140 3798194 cpu.go:275] no items to output this cycle E1121 14:38:26.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:38:26.412269 3798194 memory.go:191] Add success. I1121 14:38:26.416521 3798194 cpu.go:282] Add success. I1121 14:38:26.442049 3798194 net.go:642] Add success. I1121 14:38:26.443640 3798194 net.go:778] primary dev: ETH0 I1121 14:38:26.443654 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:38:26.443667 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:38:29.449671 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:38:29.449756 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:38:29.449784 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:38:36.412878 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:38:36.412903 3798194 memory.go:184] no items to output this cycle I1121 14:38:36.413045 3798194 cpu.go:275] no items to output this cycle E1121 14:38:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:38:46.411984 3798194 memory.go:184] no items to output this cycle I1121 14:38:46.412148 3798194 cpu.go:275] no items to output this cycle E1121 14:38:56.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:38:56.412894 3798194 memory.go:191] Add success. W1121 14:38:56.412930 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:38:56.412942 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:38:56.412945 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:38:56.417453 3798194 cpu.go:282] Add success. I1121 14:38:56.441843 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:38:56.442352 3798194 net.go:642] Add success. I1121 14:38:56.443970 3798194 net.go:778] primary dev: ETH0 I1121 14:38:56.443984 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:38:56.443999 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:38:57.452689 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:38:57.452857 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:38:57.452911 3798194 disk_worker.go:723] disk space is not compliant W1121 14:38:57.452915 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:38:57.464888 3798194 disk_worker.go:509] system disk:vda1 I1121 14:38:57.464928 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:38:57.676559 3798194 disk_info.go:125] begin check local disk info of client I1121 14:38:57.680369 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:38:57.680378 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047f400 0xc00047f440] I1121 14:38:58.446303 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:38:59.450406 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:38:59.450488 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:38:59.450513 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:38:59.471082 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:39:06.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:39:06.413248 3798194 memory.go:184] no items to output this cycle I1121 14:39:06.413391 3798194 cpu.go:275] no items to output this cycle E1121 14:39:16.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:39:16.412252 3798194 memory.go:184] no items to output this cycle I1121 14:39:16.412396 3798194 cpu.go:275] no items to output this cycle I1121 14:39:17.487447 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:39:17.487457 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:39:26.411296 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:39:26.411330 3798194 memory.go:191] Add success. I1121 14:39:26.411679 3798194 cpu.go:282] Add success. I1121 14:39:26.441304 3798194 net.go:642] Add success. I1121 14:39:26.445333 3798194 net.go:778] primary dev: ETH0 I1121 14:39:26.445347 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:39:26.445360 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:39:29.450029 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:39:29.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:39:29.450150 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:39:36.412366 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:39:36.412393 3798194 memory.go:184] no items to output this cycle I1121 14:39:36.412534 3798194 cpu.go:275] no items to output this cycle E1121 14:39:46.412788 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:39:46.412811 3798194 memory.go:184] no items to output this cycle I1121 14:39:46.412962 3798194 cpu.go:275] no items to output this cycle E1121 14:39:56.413555 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:39:56.413593 3798194 memory.go:191] Add success. W1121 14:39:56.413621 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:39:56.413632 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:39:56.413635 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:39:56.417595 3798194 cpu.go:282] Add success. I1121 14:39:56.440950 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:39:56.441532 3798194 net.go:642] Add success. I1121 14:39:56.442922 3798194 net.go:778] primary dev: ETH0 I1121 14:39:56.442936 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:39:56.442948 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:39:57.451226 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:39:57.451384 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:39:57.451435 3798194 disk_worker.go:723] disk space is not compliant W1121 14:39:57.451438 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:39:57.459780 3798194 disk_worker.go:509] system disk:vda1 I1121 14:39:57.459821 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:39:57.680468 3798194 disk_info.go:125] begin check local disk info of client I1121 14:39:57.684292 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:39:57.684301 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e6280 0xc0003e62c0] I1121 14:39:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:39:59.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:39:59.450491 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:39:59.450516 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:39:59.471040 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:40:06.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:40:06.412888 3798194 memory.go:184] no items to output this cycle I1121 14:40:06.413035 3798194 cpu.go:275] no items to output this cycle E1121 14:40:16.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:40:16.412260 3798194 memory.go:184] no items to output this cycle I1121 14:40:16.412411 3798194 cpu.go:275] no items to output this cycle E1121 14:40:26.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:40:26.413273 3798194 memory.go:191] Add success. I1121 14:40:26.417829 3798194 cpu.go:282] Add success. I1121 14:40:26.442142 3798194 net.go:642] Add success. I1121 14:40:26.443660 3798194 net.go:778] primary dev: ETH0 I1121 14:40:26.443676 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:40:26.443690 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:40:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:40:29.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:40:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:40:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:40:36.412254 3798194 memory.go:184] no items to output this cycle I1121 14:40:36.412407 3798194 cpu.go:275] no items to output this cycle E1121 14:40:46.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:40:46.412875 3798194 memory.go:184] no items to output this cycle I1121 14:40:46.413025 3798194 cpu.go:275] no items to output this cycle E1121 14:40:56.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:40:56.413280 3798194 memory.go:191] Add success. W1121 14:40:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:40:56.413323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:40:56.413327 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:40:56.418188 3798194 cpu.go:282] Add success. I1121 14:40:56.443107 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:40:56.443609 3798194 net.go:642] Add success. I1121 14:40:56.446447 3798194 net.go:778] primary dev: ETH0 I1121 14:40:56.446463 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:40:56.446474 3798194 net.go:659] Add eni metrics success, count: 7 E1121 14:40:57.445601 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 14:40:57.445616 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:40:57.445822 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:40:57.445895 3798194 disk_worker.go:723] disk space is not compliant W1121 14:40:57.445898 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:40:57.448020 3798194 disk_worker.go:509] system disk:vda1 I1121 14:40:57.448062 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:40:57.684401 3798194 disk_info.go:125] begin check local disk info of client I1121 14:40:57.688322 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:40:57.688331 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000418780 0xc0004187c0] E1121 14:40:58.448524 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 14:40:58.448538 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 14:40:59.448972 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 14:40:59.450046 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:40:59.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:40:59.450123 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:40:59.471639 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:41:06.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:41:06.411982 3798194 memory.go:184] no items to output this cycle I1121 14:41:06.412135 3798194 cpu.go:275] no items to output this cycle E1121 14:41:16.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:41:16.412267 3798194 memory.go:184] no items to output this cycle I1121 14:41:16.412415 3798194 cpu.go:275] no items to output this cycle E1121 14:41:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:41:26.411955 3798194 memory.go:191] Add success. I1121 14:41:26.412300 3798194 cpu.go:282] Add success. I1121 14:41:26.441935 3798194 net.go:642] Add success. I1121 14:41:26.445830 3798194 net.go:778] primary dev: ETH0 I1121 14:41:26.445845 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:41:26.445857 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:41:29.450386 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:41:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:41:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:41:36.412089 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:41:36.412129 3798194 memory.go:184] no items to output this cycle I1121 14:41:36.412274 3798194 cpu.go:275] no items to output this cycle E1121 14:41:46.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:41:46.412262 3798194 memory.go:184] no items to output this cycle I1121 14:41:46.412420 3798194 cpu.go:275] no items to output this cycle E1121 14:41:56.412019 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:41:56.412068 3798194 memory.go:191] Add success. W1121 14:41:56.412098 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:41:56.412123 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:41:56.412129 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:41:56.412709 3798194 cpu.go:282] Add success. I1121 14:41:56.440982 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:41:56.441578 3798194 net.go:642] Add success. I1121 14:41:56.445544 3798194 net.go:778] primary dev: ETH0 I1121 14:41:56.445559 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:41:56.445571 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:41:57.450679 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:41:57.450883 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:41:57.450896 3798194 disk_worker.go:723] disk space is not compliant W1121 14:41:57.450900 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:41:57.459207 3798194 disk_worker.go:509] system disk:vda1 I1121 14:41:57.459249 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:41:57.688433 3798194 disk_info.go:125] begin check local disk info of client I1121 14:41:57.692275 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:41:57.692284 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e140 0xc00030e180] I1121 14:41:58.446129 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:41:59.450223 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:41:59.450303 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:41:59.450328 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:41:59.470856 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:42:06.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:42:06.412266 3798194 memory.go:184] no items to output this cycle I1121 14:42:06.412412 3798194 cpu.go:275] no items to output this cycle E1121 14:42:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:42:16.413255 3798194 memory.go:184] no items to output this cycle I1121 14:42:16.413408 3798194 cpu.go:275] no items to output this cycle I1121 14:42:17.487618 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:42:17.487627 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:42:26.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:42:26.412271 3798194 memory.go:191] Add success. I1121 14:42:26.416469 3798194 cpu.go:282] Add success. I1121 14:42:26.442078 3798194 net.go:642] Add success. I1121 14:42:26.443446 3798194 net.go:778] primary dev: ETH0 I1121 14:42:26.443461 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:42:26.443475 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:42:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:42:29.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:42:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:42:36.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:42:36.413259 3798194 memory.go:184] no items to output this cycle I1121 14:42:36.413403 3798194 cpu.go:275] no items to output this cycle E1121 14:42:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:42:46.411975 3798194 memory.go:184] no items to output this cycle I1121 14:42:46.412127 3798194 cpu.go:275] no items to output this cycle E1121 14:42:56.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:42:56.411947 3798194 memory.go:191] Add success. W1121 14:42:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:42:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:42:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:42:56.412356 3798194 cpu.go:282] Add success. I1121 14:42:56.441807 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:42:56.442407 3798194 net.go:642] Add success. I1121 14:42:56.446551 3798194 net.go:778] primary dev: ETH0 I1121 14:42:56.446567 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:42:56.446581 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:42:57.450596 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:42:57.450793 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:42:57.450804 3798194 disk_worker.go:723] disk space is not compliant W1121 14:42:57.450807 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:42:57.459080 3798194 disk_worker.go:509] system disk:vda1 I1121 14:42:57.459136 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:42:57.692387 3798194 disk_info.go:125] begin check local disk info of client I1121 14:42:57.696175 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:42:57.696184 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040e0c0 0xc00040e100] I1121 14:42:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:42:59.450096 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:42:59.450182 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:42:59.450210 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:42:59.471721 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:43:06.412764 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:43:06.412786 3798194 memory.go:184] no items to output this cycle I1121 14:43:06.412948 3798194 cpu.go:275] no items to output this cycle E1121 14:43:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:43:16.412258 3798194 memory.go:184] no items to output this cycle I1121 14:43:16.412402 3798194 cpu.go:275] no items to output this cycle E1121 14:43:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:43:26.411951 3798194 memory.go:191] Add success. I1121 14:43:26.412306 3798194 cpu.go:282] Add success. I1121 14:43:26.441939 3798194 net.go:642] Add success. I1121 14:43:26.445874 3798194 net.go:778] primary dev: ETH0 I1121 14:43:26.445889 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:43:26.445903 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:43:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:43:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:43:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:43:36.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:43:36.413263 3798194 memory.go:184] no items to output this cycle I1121 14:43:36.413414 3798194 cpu.go:275] no items to output this cycle E1121 14:43:46.412870 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:43:46.412891 3798194 memory.go:184] no items to output this cycle I1121 14:43:46.413040 3798194 cpu.go:275] no items to output this cycle E1121 14:43:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:43:56.411956 3798194 memory.go:191] Add success. W1121 14:43:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:43:56.412001 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:43:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:43:56.412349 3798194 cpu.go:282] Add success. I1121 14:43:56.441808 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:43:56.442314 3798194 net.go:642] Add success. I1121 14:43:56.446733 3798194 net.go:778] primary dev: ETH0 I1121 14:43:56.446748 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:43:56.446760 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:43:57.450670 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:43:57.450834 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:43:57.450891 3798194 disk_worker.go:723] disk space is not compliant W1121 14:43:57.450894 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:43:57.459208 3798194 disk_worker.go:509] system disk:vda1 I1121 14:43:57.459251 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:43:57.696280 3798194 disk_info.go:125] begin check local disk info of client I1121 14:43:57.700062 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:43:57.700070 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003976c0 0xc000397700] I1121 14:43:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:43:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:43:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:43:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:43:59.471638 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:44:06.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:44:06.413250 3798194 memory.go:184] no items to output this cycle I1121 14:44:06.413396 3798194 cpu.go:275] no items to output this cycle E1121 14:44:16.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:44:16.412268 3798194 memory.go:184] no items to output this cycle I1121 14:44:16.412417 3798194 cpu.go:275] no items to output this cycle E1121 14:44:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:44:26.411950 3798194 memory.go:191] Add success. I1121 14:44:26.412293 3798194 cpu.go:282] Add success. I1121 14:44:26.441846 3798194 net.go:642] Add success. I1121 14:44:26.445946 3798194 net.go:778] primary dev: ETH0 I1121 14:44:26.445963 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:44:26.446078 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:44:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:44:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:44:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:44:36.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:44:36.412260 3798194 memory.go:184] no items to output this cycle I1121 14:44:36.412409 3798194 cpu.go:275] no items to output this cycle E1121 14:44:46.412344 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:44:46.412367 3798194 memory.go:184] no items to output this cycle I1121 14:44:46.412514 3798194 cpu.go:275] no items to output this cycle E1121 14:44:56.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:44:56.413274 3798194 memory.go:191] Add success. W1121 14:44:56.413309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:44:56.413321 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:44:56.413325 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:44:56.417598 3798194 cpu.go:282] Add success. I1121 14:44:56.441186 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:44:56.441656 3798194 net.go:642] Add success. I1121 14:44:56.443143 3798194 net.go:778] primary dev: ETH0 I1121 14:44:56.443157 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:44:56.443169 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:44:57.451570 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:44:57.451792 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:44:57.451803 3798194 disk_worker.go:723] disk space is not compliant W1121 14:44:57.451808 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:44:57.460142 3798194 disk_worker.go:509] system disk:vda1 I1121 14:44:57.460182 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:44:57.700155 3798194 disk_info.go:125] begin check local disk info of client I1121 14:44:57.703922 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:44:57.703930 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000403b00 0xc000403b40] I1121 14:44:58.445681 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:44:59.450598 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:44:59.450681 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:44:59.450707 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:44:59.471184 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:45:06.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:45:06.413255 3798194 memory.go:184] no items to output this cycle I1121 14:45:06.413397 3798194 cpu.go:275] no items to output this cycle E1121 14:45:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:45:16.411979 3798194 memory.go:184] no items to output this cycle I1121 14:45:16.412134 3798194 cpu.go:275] no items to output this cycle I1121 14:45:17.488284 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:45:17.488293 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:45:26.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:45:26.412896 3798194 memory.go:191] Add success. I1121 14:45:26.417143 3798194 cpu.go:282] Add success. I1121 14:45:26.441674 3798194 net.go:642] Add success. I1121 14:45:26.443096 3798194 net.go:778] primary dev: ETH0 I1121 14:45:26.443119 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:45:26.443133 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:45:29.449978 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:45:29.450053 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:45:29.450078 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:45:36.412250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:45:36.412274 3798194 memory.go:184] no items to output this cycle I1121 14:45:36.412419 3798194 cpu.go:275] no items to output this cycle E1121 14:45:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:45:46.412262 3798194 memory.go:184] no items to output this cycle I1121 14:45:46.412407 3798194 cpu.go:275] no items to output this cycle E1121 14:45:56.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:45:56.412890 3798194 memory.go:191] Add success. W1121 14:45:56.412924 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:45:56.412936 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:45:56.412940 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:45:56.417468 3798194 cpu.go:282] Add success. I1121 14:45:56.441680 3798194 event_worker.go:152] Polling the log file for events... I1121 14:45:56.445347 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:45:56.446317 3798194 net.go:642] Add success. I1121 14:45:56.447858 3798194 net.go:778] primary dev: ETH0 I1121 14:45:56.447873 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:45:56.447887 3798194 net.go:659] Add eni metrics success, count: 7 E1121 14:45:57.452743 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 14:45:57.452758 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:45:57.452918 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:45:57.452972 3798194 disk_worker.go:723] disk space is not compliant W1121 14:45:57.452975 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:45:57.461362 3798194 disk_worker.go:509] system disk:vda1 I1121 14:45:57.461404 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:45:57.704035 3798194 disk_info.go:125] begin check local disk info of client I1121 14:45:57.707847 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:45:57.707855 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004cd940 0xc0004cd980] E1121 14:45:58.448699 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 14:45:58.448714 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 14:45:59.448263 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 14:45:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:45:59.450447 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:45:59.450468 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:45:59.470991 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:46:06.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:46:06.412253 3798194 memory.go:184] no items to output this cycle I1121 14:46:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 14:46:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:46:16.412257 3798194 memory.go:184] no items to output this cycle I1121 14:46:16.412411 3798194 cpu.go:275] no items to output this cycle E1121 14:46:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:46:26.413269 3798194 memory.go:191] Add success. I1121 14:46:26.417507 3798194 cpu.go:282] Add success. I1121 14:46:26.442067 3798194 net.go:642] Add success. I1121 14:46:26.443677 3798194 net.go:778] primary dev: ETH0 I1121 14:46:26.443693 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:46:26.443706 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:46:29.450037 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:46:29.450145 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:46:29.450171 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:46:36.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:46:36.412875 3798194 memory.go:184] no items to output this cycle I1121 14:46:36.413020 3798194 cpu.go:275] no items to output this cycle E1121 14:46:46.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:46:46.412261 3798194 memory.go:184] no items to output this cycle I1121 14:46:46.412404 3798194 cpu.go:275] no items to output this cycle E1121 14:46:56.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:46:56.412274 3798194 memory.go:191] Add success. W1121 14:46:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:46:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:46:56.412318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:46:56.416559 3798194 cpu.go:282] Add success. I1121 14:46:56.441949 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:46:56.442549 3798194 net.go:642] Add success. I1121 14:46:56.444046 3798194 net.go:778] primary dev: ETH0 I1121 14:46:56.444060 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:46:56.444072 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:46:57.450368 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:46:57.450527 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:46:57.450579 3798194 disk_worker.go:723] disk space is not compliant W1121 14:46:57.450582 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:46:57.459007 3798194 disk_worker.go:509] system disk:vda1 I1121 14:46:57.459052 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:46:57.708135 3798194 disk_info.go:125] begin check local disk info of client I1121 14:46:57.711907 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:46:57.711916 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040c940 0xc00040c980] I1121 14:46:58.446673 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:46:59.449662 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:46:59.449740 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:46:59.449768 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:46:59.471367 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:47:06.413299 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:47:06.413320 3798194 memory.go:184] no items to output this cycle I1121 14:47:06.413674 3798194 cpu.go:275] no items to output this cycle E1121 14:47:16.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:47:16.412259 3798194 memory.go:184] no items to output this cycle I1121 14:47:16.412407 3798194 cpu.go:275] no items to output this cycle E1121 14:47:26.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:47:26.412284 3798194 memory.go:191] Add success. I1121 14:47:26.416588 3798194 cpu.go:282] Add success. I1121 14:47:26.442170 3798194 net.go:642] Add success. I1121 14:47:26.443756 3798194 net.go:778] primary dev: ETH0 I1121 14:47:26.443771 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:47:26.443786 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:47:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:47:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:47:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:47:36.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:47:36.412262 3798194 memory.go:184] no items to output this cycle I1121 14:47:36.412408 3798194 cpu.go:275] no items to output this cycle E1121 14:47:46.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:47:46.412885 3798194 memory.go:184] no items to output this cycle I1121 14:47:46.413043 3798194 cpu.go:275] no items to output this cycle E1121 14:47:56.412879 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:47:56.412918 3798194 memory.go:191] Add success. W1121 14:47:56.412952 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:47:56.412965 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:47:56.412968 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:47:56.417447 3798194 cpu.go:282] Add success. I1121 14:47:56.441868 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:47:56.442366 3798194 net.go:642] Add success. I1121 14:47:56.444097 3798194 net.go:778] primary dev: ETH0 I1121 14:47:56.444125 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:47:56.444141 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:47:57.450974 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:47:57.451212 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:47:57.451224 3798194 disk_worker.go:723] disk space is not compliant W1121 14:47:57.451227 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:47:57.459978 3798194 disk_worker.go:509] system disk:vda1 I1121 14:47:57.460017 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:47:57.712020 3798194 disk_info.go:125] begin check local disk info of client I1121 14:47:57.715765 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:47:57.715773 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037ee80 0xc00037eec0] I1121 14:47:58.446389 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:47:59.450500 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:47:59.450583 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:47:59.450609 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:47:59.471127 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:48:06.413202 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:48:06.413224 3798194 memory.go:184] no items to output this cycle I1121 14:48:06.413369 3798194 cpu.go:275] no items to output this cycle E1121 14:48:16.413212 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:48:16.413235 3798194 memory.go:184] no items to output this cycle I1121 14:48:16.413385 3798194 cpu.go:275] no items to output this cycle I1121 14:48:17.488418 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:48:17.488426 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:48:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:48:26.413285 3798194 memory.go:191] Add success. I1121 14:48:26.417992 3798194 cpu.go:282] Add success. I1121 14:48:26.441523 3798194 net.go:642] Add success. I1121 14:48:26.443010 3798194 net.go:778] primary dev: ETH0 I1121 14:48:26.443025 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:48:26.443038 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:48:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:48:29.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:48:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:48:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:48:36.411977 3798194 memory.go:184] no items to output this cycle I1121 14:48:36.412351 3798194 cpu.go:275] no items to output this cycle E1121 14:48:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:48:46.411970 3798194 memory.go:184] no items to output this cycle I1121 14:48:46.412130 3798194 cpu.go:275] no items to output this cycle E1121 14:48:56.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:48:56.412264 3798194 memory.go:191] Add success. W1121 14:48:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:48:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:48:56.412313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:48:56.416795 3798194 cpu.go:282] Add success. I1121 14:48:56.441164 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:48:56.441642 3798194 net.go:642] Add success. I1121 14:48:56.443080 3798194 net.go:778] primary dev: ETH0 I1121 14:48:56.443095 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:48:56.443118 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:48:57.451017 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:48:57.451246 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:48:57.451257 3798194 disk_worker.go:723] disk space is not compliant W1121 14:48:57.451260 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:48:57.459990 3798194 disk_worker.go:509] system disk:vda1 I1121 14:48:57.460033 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:48:57.715877 3798194 disk_info.go:125] begin check local disk info of client I1121 14:48:57.719722 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:48:57.719732 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041ed40 0xc00041ed80] I1121 14:48:58.446377 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:48:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:48:59.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:48:59.450509 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:48:59.471035 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:49:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:49:06.411987 3798194 memory.go:184] no items to output this cycle I1121 14:49:06.412143 3798194 cpu.go:275] no items to output this cycle E1121 14:49:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:49:16.411980 3798194 memory.go:184] no items to output this cycle I1121 14:49:16.412136 3798194 cpu.go:275] no items to output this cycle E1121 14:49:26.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:49:26.412274 3798194 memory.go:191] Add success. I1121 14:49:26.416346 3798194 cpu.go:282] Add success. I1121 14:49:26.441897 3798194 net.go:642] Add success. I1121 14:49:26.443324 3798194 net.go:778] primary dev: ETH0 I1121 14:49:26.443338 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:49:26.443352 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:49:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:49:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:49:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:49:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:49:36.411979 3798194 memory.go:184] no items to output this cycle I1121 14:49:36.412135 3798194 cpu.go:275] no items to output this cycle E1121 14:49:46.412768 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:49:46.412791 3798194 memory.go:184] no items to output this cycle I1121 14:49:46.412949 3798194 cpu.go:275] no items to output this cycle E1121 14:49:56.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:49:56.412276 3798194 memory.go:191] Add success. W1121 14:49:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:49:56.412327 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:49:56.412330 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:49:56.416835 3798194 cpu.go:282] Add success. I1121 14:49:56.441249 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:49:56.441726 3798194 net.go:642] Add success. I1121 14:49:56.443292 3798194 net.go:778] primary dev: ETH0 I1121 14:49:56.443307 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:49:56.443318 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:49:57.451331 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:49:57.451502 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:49:57.451559 3798194 disk_worker.go:723] disk space is not compliant W1121 14:49:57.451562 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:49:57.459888 3798194 disk_worker.go:509] system disk:vda1 I1121 14:49:57.459928 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:49:57.720151 3798194 disk_info.go:125] begin check local disk info of client I1121 14:49:57.723868 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:49:57.723876 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000479e80 0xc000479ec0] I1121 14:49:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:49:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:49:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:49:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:49:59.471675 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:50:06.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:50:06.411993 3798194 memory.go:184] no items to output this cycle I1121 14:50:06.412151 3798194 cpu.go:275] no items to output this cycle E1121 14:50:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:50:16.411982 3798194 memory.go:184] no items to output this cycle I1121 14:50:16.412154 3798194 cpu.go:275] no items to output this cycle E1121 14:50:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:50:26.411955 3798194 memory.go:191] Add success. I1121 14:50:26.412296 3798194 cpu.go:282] Add success. I1121 14:50:26.441929 3798194 net.go:642] Add success. I1121 14:50:26.445949 3798194 net.go:778] primary dev: ETH0 I1121 14:50:26.445965 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:50:26.445979 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:50:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:50:29.450088 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:50:29.450121 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:50:36.412422 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:50:36.412453 3798194 memory.go:184] no items to output this cycle I1121 14:50:36.412661 3798194 cpu.go:275] no items to output this cycle E1121 14:50:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:50:46.411975 3798194 memory.go:184] no items to output this cycle I1121 14:50:46.412133 3798194 cpu.go:275] no items to output this cycle E1121 14:50:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:50:56.412264 3798194 memory.go:191] Add success. W1121 14:50:56.412291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:50:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:50:56.412306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:50:56.416415 3798194 cpu.go:282] Add success. I1121 14:50:56.444875 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:50:56.445362 3798194 net.go:642] Add success. I1121 14:50:56.447002 3798194 net.go:778] primary dev: ETH0 I1121 14:50:56.447017 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:50:56.447029 3798194 net.go:659] Add eni metrics success, count: 7 E1121 14:50:57.451003 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 14:50:57.451018 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:50:57.451238 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:50:57.451248 3798194 disk_worker.go:723] disk space is not compliant W1121 14:50:57.451251 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:50:57.460081 3798194 disk_worker.go:509] system disk:vda1 I1121 14:50:57.460231 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:50:57.723967 3798194 disk_info.go:125] begin check local disk info of client I1121 14:50:57.727793 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:50:57.727802 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040b1c0 0xc00040b200] E1121 14:50:58.448445 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 14:50:58.448459 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 14:50:59.448974 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 14:50:59.450050 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:50:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:50:59.450127 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:50:59.471649 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:51:06.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:51:06.412886 3798194 memory.go:184] no items to output this cycle I1121 14:51:06.413039 3798194 cpu.go:275] no items to output this cycle E1121 14:51:16.413351 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:51:16.413487 3798194 memory.go:184] no items to output this cycle I1121 14:51:16.413775 3798194 cpu.go:275] no items to output this cycle I1121 14:51:17.488577 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:51:17.488586 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:51:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:51:26.412269 3798194 memory.go:191] Add success. I1121 14:51:26.416409 3798194 cpu.go:282] Add success. I1121 14:51:26.441970 3798194 net.go:642] Add success. I1121 14:51:26.443366 3798194 net.go:778] primary dev: ETH0 I1121 14:51:26.443380 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:51:26.443393 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:51:29.450129 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:51:29.450209 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:51:29.450238 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:51:36.412288 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:51:36.412318 3798194 memory.go:184] no items to output this cycle I1121 14:51:36.412525 3798194 cpu.go:275] no items to output this cycle E1121 14:51:46.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:51:46.412889 3798194 memory.go:184] no items to output this cycle I1121 14:51:46.413038 3798194 cpu.go:275] no items to output this cycle E1121 14:51:56.413001 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:51:56.413037 3798194 memory.go:191] Add success. W1121 14:51:56.413065 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:51:56.413083 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:51:56.413087 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:51:56.417313 3798194 cpu.go:282] Add success. I1121 14:51:56.441719 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:51:56.442251 3798194 net.go:642] Add success. I1121 14:51:56.443806 3798194 net.go:778] primary dev: ETH0 I1121 14:51:56.443822 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:51:56.443834 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:51:57.451286 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:51:57.451443 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:51:57.451493 3798194 disk_worker.go:723] disk space is not compliant W1121 14:51:57.451497 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:51:57.459934 3798194 disk_worker.go:509] system disk:vda1 I1121 14:51:57.459977 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:51:57.727906 3798194 disk_info.go:125] begin check local disk info of client I1121 14:51:57.731794 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:51:57.731806 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000303f40 0xc000362000] I1121 14:51:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:51:59.449836 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:51:59.449947 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:51:59.449972 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:51:59.471530 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:52:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:52:06.411984 3798194 memory.go:184] no items to output this cycle I1121 14:52:06.412143 3798194 cpu.go:275] no items to output this cycle E1121 14:52:16.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:52:16.412879 3798194 memory.go:184] no items to output this cycle I1121 14:52:16.413032 3798194 cpu.go:275] no items to output this cycle E1121 14:52:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:52:26.411954 3798194 memory.go:191] Add success. I1121 14:52:26.412304 3798194 cpu.go:282] Add success. I1121 14:52:26.441760 3798194 net.go:642] Add success. I1121 14:52:26.445970 3798194 net.go:778] primary dev: ETH0 I1121 14:52:26.445986 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:52:26.446000 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:52:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:52:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:52:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:52:36.412082 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:52:36.412104 3798194 memory.go:184] no items to output this cycle I1121 14:52:36.412269 3798194 cpu.go:275] no items to output this cycle E1121 14:52:46.412377 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:52:46.412405 3798194 memory.go:184] no items to output this cycle I1121 14:52:46.412553 3798194 cpu.go:275] no items to output this cycle E1121 14:52:56.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:52:56.412276 3798194 memory.go:191] Add success. W1121 14:52:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:52:56.412319 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:52:56.412323 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:52:56.416652 3798194 cpu.go:282] Add success. I1121 14:52:56.440968 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:52:56.441565 3798194 net.go:642] Add success. I1121 14:52:56.443257 3798194 net.go:778] primary dev: ETH0 I1121 14:52:56.443272 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:52:56.443285 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:52:57.451761 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:52:57.451967 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:52:57.451978 3798194 disk_worker.go:723] disk space is not compliant W1121 14:52:57.451981 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:52:57.460950 3798194 disk_worker.go:509] system disk:vda1 I1121 14:52:57.460991 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:52:57.732145 3798194 disk_info.go:125] begin check local disk info of client I1121 14:52:57.735994 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:52:57.736003 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000362140 0xc000362180] I1121 14:52:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:52:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:52:59.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:52:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:52:59.471680 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:53:06.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:53:06.411990 3798194 memory.go:184] no items to output this cycle I1121 14:53:06.412151 3798194 cpu.go:275] no items to output this cycle E1121 14:53:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:53:16.411972 3798194 memory.go:184] no items to output this cycle I1121 14:53:16.412129 3798194 cpu.go:275] no items to output this cycle E1121 14:53:26.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:53:26.412275 3798194 memory.go:191] Add success. I1121 14:53:26.416590 3798194 cpu.go:282] Add success. I1121 14:53:26.442038 3798194 net.go:642] Add success. I1121 14:53:26.443601 3798194 net.go:778] primary dev: ETH0 I1121 14:53:26.443616 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:53:26.443630 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:53:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:53:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:53:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:53:36.412872 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:53:36.412893 3798194 memory.go:184] no items to output this cycle I1121 14:53:36.413038 3798194 cpu.go:275] no items to output this cycle E1121 14:53:46.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:53:46.412260 3798194 memory.go:184] no items to output this cycle I1121 14:53:46.412410 3798194 cpu.go:275] no items to output this cycle E1121 14:53:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:53:56.412268 3798194 memory.go:191] Add success. W1121 14:53:56.412295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:53:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:53:56.412310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:53:56.416890 3798194 cpu.go:282] Add success. I1121 14:53:56.441300 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:53:56.441774 3798194 net.go:642] Add success. I1121 14:53:56.443415 3798194 net.go:778] primary dev: ETH0 I1121 14:53:56.443431 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:53:56.443445 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:53:57.451239 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:53:57.451441 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:53:57.451452 3798194 disk_worker.go:723] disk space is not compliant W1121 14:53:57.451455 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:53:57.460558 3798194 disk_worker.go:509] system disk:vda1 I1121 14:53:57.460596 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:53:57.736106 3798194 disk_info.go:125] begin check local disk info of client I1121 14:53:57.740033 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:53:57.740043 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00049d440 0xc00049d480] I1121 14:53:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:53:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:53:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:53:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:53:59.471652 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:54:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:54:06.412268 3798194 memory.go:184] no items to output this cycle I1121 14:54:06.412417 3798194 cpu.go:275] no items to output this cycle E1121 14:54:16.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:54:16.413268 3798194 memory.go:184] no items to output this cycle I1121 14:54:16.413413 3798194 cpu.go:275] no items to output this cycle I1121 14:54:17.489671 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:54:17.489680 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:54:26.412223 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:54:26.412258 3798194 memory.go:191] Add success. I1121 14:54:26.416883 3798194 cpu.go:282] Add success. I1121 14:54:26.441436 3798194 net.go:642] Add success. I1121 14:54:26.443122 3798194 net.go:778] primary dev: ETH0 I1121 14:54:26.443139 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:54:26.443152 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:54:29.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:54:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:54:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:54:36.411759 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:54:36.411784 3798194 memory.go:184] no items to output this cycle I1121 14:54:36.411926 3798194 cpu.go:275] no items to output this cycle E1121 14:54:46.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:54:46.413262 3798194 memory.go:184] no items to output this cycle I1121 14:54:46.413411 3798194 cpu.go:275] no items to output this cycle E1121 14:54:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:54:56.411948 3798194 memory.go:191] Add success. W1121 14:54:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:54:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:54:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:54:56.412359 3798194 cpu.go:282] Add success. I1121 14:54:56.441839 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:54:56.442455 3798194 net.go:642] Add success. I1121 14:54:56.446500 3798194 net.go:778] primary dev: ETH0 I1121 14:54:56.446514 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:54:56.446526 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:54:57.450980 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:54:57.451223 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:54:57.451235 3798194 disk_worker.go:723] disk space is not compliant W1121 14:54:57.451238 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:54:57.462664 3798194 disk_worker.go:509] system disk:vda1 I1121 14:54:57.462707 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:54:57.741135 3798194 disk_info.go:125] begin check local disk info of client I1121 14:54:57.744971 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:54:57.744980 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d8200 0xc0003d8240] I1121 14:54:58.446584 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:54:59.449639 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:54:59.449713 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:54:59.449736 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:54:59.471287 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:55:06.412975 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:55:06.412997 3798194 memory.go:184] no items to output this cycle I1121 14:55:06.413157 3798194 cpu.go:275] no items to output this cycle E1121 14:55:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:55:16.412256 3798194 memory.go:184] no items to output this cycle I1121 14:55:16.412409 3798194 cpu.go:275] no items to output this cycle E1121 14:55:26.413002 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:55:26.413035 3798194 memory.go:191] Add success. I1121 14:55:26.417107 3798194 cpu.go:282] Add success. I1121 14:55:26.441975 3798194 net.go:642] Add success. I1121 14:55:26.443614 3798194 net.go:778] primary dev: ETH0 I1121 14:55:26.443631 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:55:26.443650 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:55:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:55:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:55:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:55:36.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:55:36.413253 3798194 memory.go:184] no items to output this cycle I1121 14:55:36.413396 3798194 cpu.go:275] no items to output this cycle E1121 14:55:46.411160 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:55:46.411184 3798194 memory.go:184] no items to output this cycle I1121 14:55:46.411335 3798194 cpu.go:275] no items to output this cycle E1121 14:55:56.411908 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:55:56.411939 3798194 memory.go:191] Add success. W1121 14:55:56.411966 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:55:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:55:56.411981 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:55:56.412334 3798194 cpu.go:282] Add success. I1121 14:55:56.441536 3798194 event_worker.go:152] Polling the log file for events... I1121 14:55:56.444738 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:55:56.445240 3798194 net.go:642] Add success. I1121 14:55:56.449620 3798194 net.go:778] primary dev: ETH0 I1121 14:55:56.449635 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:55:56.449649 3798194 net.go:659] Add eni metrics success, count: 7 E1121 14:55:57.450967 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 14:55:57.450981 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:55:57.451168 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:55:57.451220 3798194 disk_worker.go:723] disk space is not compliant W1121 14:55:57.451223 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:55:57.460134 3798194 disk_worker.go:509] system disk:vda1 I1121 14:55:57.460178 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:55:57.745080 3798194 disk_info.go:125] begin check local disk info of client I1121 14:55:57.748916 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:55:57.748925 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00045c6c0 0xc00045c700] E1121 14:55:58.448600 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 14:55:58.448615 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 14:55:59.448618 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 14:55:59.449694 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:55:59.449746 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:55:59.449765 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:55:59.471273 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:56:06.412773 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:56:06.412795 3798194 memory.go:184] no items to output this cycle I1121 14:56:06.412950 3798194 cpu.go:275] no items to output this cycle E1121 14:56:16.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:56:16.412882 3798194 memory.go:184] no items to output this cycle I1121 14:56:16.413025 3798194 cpu.go:275] no items to output this cycle E1121 14:56:26.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:56:26.412270 3798194 memory.go:191] Add success. I1121 14:56:26.416684 3798194 cpu.go:282] Add success. I1121 14:56:26.442223 3798194 net.go:642] Add success. I1121 14:56:26.443768 3798194 net.go:778] primary dev: ETH0 I1121 14:56:26.443783 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:56:26.443795 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:56:29.450403 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:56:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:56:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:56:36.412533 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:56:36.412554 3798194 memory.go:184] no items to output this cycle I1121 14:56:36.412695 3798194 cpu.go:275] no items to output this cycle E1121 14:56:46.413252 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:56:46.413276 3798194 memory.go:184] no items to output this cycle I1121 14:56:46.413425 3798194 cpu.go:275] no items to output this cycle E1121 14:56:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:56:56.411962 3798194 memory.go:191] Add success. W1121 14:56:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:56:56.412007 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:56:56.412010 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:56:56.412384 3798194 cpu.go:282] Add success. I1121 14:56:56.440984 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:56:56.441574 3798194 net.go:642] Add success. I1121 14:56:56.445328 3798194 net.go:778] primary dev: ETH0 I1121 14:56:56.445342 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:56:56.445356 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:56:57.451282 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:56:57.451500 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:56:57.451510 3798194 disk_worker.go:723] disk space is not compliant W1121 14:56:57.451513 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:56:57.464145 3798194 disk_worker.go:509] system disk:vda1 I1121 14:56:57.464201 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:56:57.749148 3798194 disk_info.go:125] begin check local disk info of client I1121 14:56:57.753001 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:56:57.753012 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ea00 0xc00030ea40] I1121 14:56:58.446385 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:56:59.449965 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:56:59.450042 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:56:59.450071 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:56:59.471569 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:57:06.412770 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:57:06.412792 3798194 memory.go:184] no items to output this cycle I1121 14:57:06.412943 3798194 cpu.go:275] no items to output this cycle E1121 14:57:16.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:57:16.412253 3798194 memory.go:184] no items to output this cycle I1121 14:57:16.412401 3798194 cpu.go:275] no items to output this cycle I1121 14:57:17.490540 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 14:57:17.490550 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 14:57:26.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:57:26.412275 3798194 memory.go:191] Add success. I1121 14:57:26.416488 3798194 cpu.go:282] Add success. I1121 14:57:26.442031 3798194 net.go:642] Add success. I1121 14:57:26.443330 3798194 net.go:778] primary dev: ETH0 I1121 14:57:26.443346 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:57:26.443366 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:57:29.450385 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:57:29.450463 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:57:29.450491 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:57:36.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:57:36.411983 3798194 memory.go:184] no items to output this cycle I1121 14:57:36.412132 3798194 cpu.go:275] no items to output this cycle E1121 14:57:46.413163 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:57:46.413203 3798194 memory.go:184] no items to output this cycle I1121 14:57:46.413404 3798194 cpu.go:275] no items to output this cycle E1121 14:57:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:57:56.411951 3798194 memory.go:191] Add success. W1121 14:57:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:57:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:57:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:57:56.412334 3798194 cpu.go:282] Add success. I1121 14:57:56.441819 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:57:56.442333 3798194 net.go:642] Add success. I1121 14:57:56.446314 3798194 net.go:778] primary dev: ETH0 I1121 14:57:56.446329 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:57:56.446342 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:57:57.450969 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:57:57.451204 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:57:57.451216 3798194 disk_worker.go:723] disk space is not compliant W1121 14:57:57.451219 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:57:57.459735 3798194 disk_worker.go:509] system disk:vda1 I1121 14:57:57.459778 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:57:57.753147 3798194 disk_info.go:125] begin check local disk info of client I1121 14:57:57.756972 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:57:57.756980 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000456cc0 0xc000456d00] I1121 14:57:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:57:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:57:59.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:57:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:57:59.471657 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:58:06.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:58:06.412257 3798194 memory.go:184] no items to output this cycle I1121 14:58:06.412396 3798194 cpu.go:275] no items to output this cycle E1121 14:58:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:58:16.413265 3798194 memory.go:184] no items to output this cycle I1121 14:58:16.413423 3798194 cpu.go:275] no items to output this cycle E1121 14:58:26.412870 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:58:26.412906 3798194 memory.go:191] Add success. I1121 14:58:26.417105 3798194 cpu.go:282] Add success. I1121 14:58:26.441635 3798194 net.go:642] Add success. I1121 14:58:26.443063 3798194 net.go:778] primary dev: ETH0 I1121 14:58:26.443077 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:58:26.443089 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:58:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:58:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:58:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:58:36.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:58:36.413261 3798194 memory.go:184] no items to output this cycle I1121 14:58:36.413405 3798194 cpu.go:275] no items to output this cycle E1121 14:58:46.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:58:46.412259 3798194 memory.go:184] no items to output this cycle I1121 14:58:46.412412 3798194 cpu.go:275] no items to output this cycle E1121 14:58:56.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:58:56.412888 3798194 memory.go:191] Add success. W1121 14:58:56.412922 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:58:56.412933 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:58:56.412938 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:58:56.422734 3798194 cpu.go:282] Add success. I1121 14:58:56.441590 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:58:56.442071 3798194 net.go:642] Add success. I1121 14:58:56.443564 3798194 net.go:778] primary dev: ETH0 I1121 14:58:56.443579 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:58:56.443594 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:58:57.452170 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:58:57.452333 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:58:57.452387 3798194 disk_worker.go:723] disk space is not compliant W1121 14:58:57.452390 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:58:57.464428 3798194 disk_worker.go:509] system disk:vda1 I1121 14:58:57.464482 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:58:57.757084 3798194 disk_info.go:125] begin check local disk info of client I1121 14:58:57.761020 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:58:57.761029 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037c2c0 0xc00037c300] I1121 14:58:58.446357 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:58:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:58:59.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:58:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:58:59.471644 3798194 disk_local_worker.go:436] Get disk info: [] E1121 14:59:06.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:59:06.413257 3798194 memory.go:184] no items to output this cycle I1121 14:59:06.413397 3798194 cpu.go:275] no items to output this cycle E1121 14:59:16.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:59:16.413251 3798194 memory.go:184] no items to output this cycle I1121 14:59:16.413395 3798194 cpu.go:275] no items to output this cycle E1121 14:59:26.411910 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:59:26.411942 3798194 memory.go:191] Add success. I1121 14:59:26.412309 3798194 cpu.go:282] Add success. I1121 14:59:26.442067 3798194 net.go:642] Add success. I1121 14:59:26.445999 3798194 net.go:778] primary dev: ETH0 I1121 14:59:26.446013 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:59:26.446026 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:59:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:59:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:59:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 14:59:36.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:59:36.412251 3798194 memory.go:184] no items to output this cycle I1121 14:59:36.412400 3798194 cpu.go:275] no items to output this cycle E1121 14:59:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:59:46.413269 3798194 memory.go:184] no items to output this cycle I1121 14:59:46.413419 3798194 cpu.go:275] no items to output this cycle E1121 14:59:56.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 14:59:56.411959 3798194 memory.go:191] Add success. W1121 14:59:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 14:59:56.412003 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 14:59:56.412007 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 14:59:56.412384 3798194 cpu.go:282] Add success. I1121 14:59:56.441840 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 14:59:56.442354 3798194 net.go:642] Add success. I1121 14:59:56.446953 3798194 net.go:778] primary dev: ETH0 I1121 14:59:56.446969 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 14:59:56.446982 3798194 net.go:659] Add eni metrics success, count: 7 I1121 14:59:57.451449 3798194 custom_config.go:106] query custom config with name: gpu W1121 14:59:57.451664 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 14:59:57.451674 3798194 disk_worker.go:723] disk space is not compliant W1121 14:59:57.451677 3798194 disk_worker.go:743] disk inode is not compliant I1121 14:59:57.460184 3798194 disk_worker.go:509] system disk:vda1 I1121 14:59:57.460227 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 14:59:57.761168 3798194 disk_info.go:125] begin check local disk info of client I1121 14:59:57.765019 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 14:59:57.765029 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00042c4c0 0xc00042c500] I1121 14:59:58.446008 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 14:59:59.450040 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 14:59:59.450122 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 14:59:59.450146 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 14:59:59.471679 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:00:06.412765 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:00:06.412787 3798194 memory.go:184] no items to output this cycle I1121 15:00:06.412950 3798194 cpu.go:275] no items to output this cycle E1121 15:00:16.412878 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:00:16.412900 3798194 memory.go:184] no items to output this cycle I1121 15:00:16.413048 3798194 cpu.go:275] no items to output this cycle I1121 15:00:17.491279 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:00:17.491287 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:00:26.411313 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:00:26.411351 3798194 memory.go:191] Add success. I1121 15:00:26.411683 3798194 cpu.go:282] Add success. I1121 15:00:26.441251 3798194 net.go:642] Add success. I1121 15:00:26.445508 3798194 net.go:778] primary dev: ETH0 I1121 15:00:26.445525 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:00:26.445542 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:00:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:00:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:00:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:00:36.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:00:36.413249 3798194 memory.go:184] no items to output this cycle I1121 15:00:36.413393 3798194 cpu.go:275] no items to output this cycle E1121 15:00:46.413184 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:00:46.413206 3798194 memory.go:184] no items to output this cycle I1121 15:00:46.413352 3798194 cpu.go:275] no items to output this cycle E1121 15:00:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:00:56.411955 3798194 memory.go:191] Add success. W1121 15:00:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:00:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:00:56.412002 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:00:56.412339 3798194 cpu.go:282] Add success. I1121 15:00:56.445633 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:00:56.446164 3798194 net.go:642] Add success. I1121 15:00:56.450525 3798194 net.go:778] primary dev: ETH0 I1121 15:00:56.450540 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:00:56.450553 3798194 net.go:659] Add eni metrics success, count: 7 E1121 15:00:57.446421 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 15:00:57.446436 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:00:57.446636 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:00:57.446709 3798194 disk_worker.go:723] disk space is not compliant W1121 15:00:57.446712 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:00:57.449472 3798194 disk_worker.go:509] system disk:vda1 I1121 15:00:57.449510 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:00:57.765141 3798194 disk_info.go:125] begin check local disk info of client I1121 15:00:57.769046 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:00:57.769056 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00049cbc0 0xc00049cc00] E1121 15:00:58.448496 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 15:00:58.448511 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 15:00:59.448983 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 15:00:59.450057 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:00:59.450124 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:00:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:00:59.471663 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:01:06.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:01:06.413262 3798194 memory.go:184] no items to output this cycle I1121 15:01:06.413414 3798194 cpu.go:275] no items to output this cycle E1121 15:01:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:01:16.411979 3798194 memory.go:184] no items to output this cycle I1121 15:01:16.412133 3798194 cpu.go:275] no items to output this cycle E1121 15:01:26.412834 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:01:26.412870 3798194 memory.go:191] Add success. I1121 15:01:26.417341 3798194 cpu.go:282] Add success. I1121 15:01:26.442012 3798194 net.go:642] Add success. I1121 15:01:26.443674 3798194 net.go:778] primary dev: ETH0 I1121 15:01:26.443688 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:01:26.443701 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:01:29.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:01:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:01:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:01:36.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:01:36.413251 3798194 memory.go:184] no items to output this cycle I1121 15:01:36.413393 3798194 cpu.go:275] no items to output this cycle E1121 15:01:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:01:46.411983 3798194 memory.go:184] no items to output this cycle I1121 15:01:46.412143 3798194 cpu.go:275] no items to output this cycle E1121 15:01:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:01:56.411956 3798194 memory.go:191] Add success. W1121 15:01:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:01:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:01:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:01:56.412360 3798194 cpu.go:282] Add success. I1121 15:01:56.441938 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:01:56.442547 3798194 net.go:642] Add success. I1121 15:01:56.446741 3798194 net.go:778] primary dev: ETH0 I1121 15:01:56.446756 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:01:56.446769 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:01:57.450623 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:01:57.450832 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:01:57.450844 3798194 disk_worker.go:723] disk space is not compliant W1121 15:01:57.450847 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:01:57.459222 3798194 disk_worker.go:509] system disk:vda1 I1121 15:01:57.459274 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:01:57.770136 3798194 disk_info.go:125] begin check local disk info of client I1121 15:01:57.773969 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:01:57.773978 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a5940 0xc0003a5980] I1121 15:01:58.446383 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:01:59.450485 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:01:59.450560 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:01:59.450584 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:01:59.471115 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:02:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:02:06.411982 3798194 memory.go:184] no items to output this cycle I1121 15:02:06.412139 3798194 cpu.go:275] no items to output this cycle E1121 15:02:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:02:16.411979 3798194 memory.go:184] no items to output this cycle I1121 15:02:16.412140 3798194 cpu.go:275] no items to output this cycle E1121 15:02:26.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:02:26.413271 3798194 memory.go:191] Add success. I1121 15:02:26.417478 3798194 cpu.go:282] Add success. I1121 15:02:26.442250 3798194 net.go:642] Add success. I1121 15:02:26.443821 3798194 net.go:778] primary dev: ETH0 I1121 15:02:26.443836 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:02:26.443856 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:02:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:02:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:02:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:02:36.413249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:02:36.413273 3798194 memory.go:184] no items to output this cycle I1121 15:02:36.413418 3798194 cpu.go:275] no items to output this cycle E1121 15:02:46.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:02:46.413257 3798194 memory.go:184] no items to output this cycle I1121 15:02:46.413414 3798194 cpu.go:275] no items to output this cycle E1121 15:02:56.412157 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:02:56.412191 3798194 memory.go:191] Add success. W1121 15:02:56.412226 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:02:56.412238 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:02:56.412241 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:02:56.416498 3798194 cpu.go:282] Add success. I1121 15:02:56.441908 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:02:56.442425 3798194 net.go:642] Add success. I1121 15:02:56.444077 3798194 net.go:778] primary dev: ETH0 I1121 15:02:56.444093 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:02:56.444123 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:02:57.453042 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:02:57.453228 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:02:57.453286 3798194 disk_worker.go:723] disk space is not compliant W1121 15:02:57.453290 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:02:57.465004 3798194 disk_worker.go:509] system disk:vda1 I1121 15:02:57.465049 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:02:57.774144 3798194 disk_info.go:125] begin check local disk info of client I1121 15:02:57.777975 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:02:57.777983 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040fd80 0xc00040fdc0] I1121 15:02:58.446385 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:02:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:02:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:02:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:02:59.471705 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:03:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:03:06.411980 3798194 memory.go:184] no items to output this cycle I1121 15:03:06.412135 3798194 cpu.go:275] no items to output this cycle E1121 15:03:16.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:03:16.411971 3798194 memory.go:184] no items to output this cycle I1121 15:03:16.412125 3798194 cpu.go:275] no items to output this cycle I1121 15:03:17.492270 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:03:17.492279 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:03:26.413292 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:03:26.413326 3798194 memory.go:191] Add success. I1121 15:03:26.417315 3798194 cpu.go:282] Add success. I1121 15:03:26.441644 3798194 net.go:642] Add success. I1121 15:03:26.443131 3798194 net.go:778] primary dev: ETH0 I1121 15:03:26.443145 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:03:26.443159 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:03:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:03:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:03:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:03:36.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:03:36.411983 3798194 memory.go:184] no items to output this cycle I1121 15:03:36.412135 3798194 cpu.go:275] no items to output this cycle E1121 15:03:46.412791 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:03:46.412815 3798194 memory.go:184] no items to output this cycle I1121 15:03:46.412963 3798194 cpu.go:275] no items to output this cycle E1121 15:03:56.412729 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:03:56.412763 3798194 memory.go:191] Add success. W1121 15:03:56.412792 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:03:56.412803 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:03:56.412806 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:03:56.417483 3798194 cpu.go:282] Add success. I1121 15:03:56.441924 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:03:56.442682 3798194 net.go:642] Add success. I1121 15:03:56.444577 3798194 net.go:778] primary dev: ETH0 I1121 15:03:56.444593 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:03:56.444611 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:03:57.450675 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:03:57.450831 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:03:57.450884 3798194 disk_worker.go:723] disk space is not compliant W1121 15:03:57.450887 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:03:57.459267 3798194 disk_worker.go:509] system disk:vda1 I1121 15:03:57.459307 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:03:57.778095 3798194 disk_info.go:125] begin check local disk info of client I1121 15:03:57.781943 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:03:57.781951 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004e18c0 0xc0004e1900] I1121 15:03:58.446370 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:03:59.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:03:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:03:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:03:59.471660 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:04:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:04:06.411979 3798194 memory.go:184] no items to output this cycle I1121 15:04:06.412137 3798194 cpu.go:275] no items to output this cycle E1121 15:04:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:04:16.411989 3798194 memory.go:184] no items to output this cycle I1121 15:04:16.412148 3798194 cpu.go:275] no items to output this cycle E1121 15:04:26.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:04:26.412896 3798194 memory.go:191] Add success. I1121 15:04:26.417076 3798194 cpu.go:282] Add success. I1121 15:04:26.441531 3798194 net.go:642] Add success. I1121 15:04:26.443011 3798194 net.go:778] primary dev: ETH0 I1121 15:04:26.443027 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:04:26.443041 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:04:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:04:29.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:04:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:04:36.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:04:36.413259 3798194 memory.go:184] no items to output this cycle I1121 15:04:36.413403 3798194 cpu.go:275] no items to output this cycle E1121 15:04:46.412261 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:04:46.412288 3798194 memory.go:184] no items to output this cycle I1121 15:04:46.412447 3798194 cpu.go:275] no items to output this cycle E1121 15:04:56.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:04:56.412891 3798194 memory.go:191] Add success. W1121 15:04:56.412925 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:04:56.412937 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:04:56.412941 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:04:56.417293 3798194 cpu.go:282] Add success. I1121 15:04:56.441687 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:04:56.442190 3798194 net.go:642] Add success. I1121 15:04:56.443821 3798194 net.go:778] primary dev: ETH0 I1121 15:04:56.443838 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:04:56.443851 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:04:57.453059 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:04:57.453294 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:04:57.453305 3798194 disk_worker.go:723] disk space is not compliant W1121 15:04:57.453308 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:04:57.464074 3798194 disk_worker.go:509] system disk:vda1 I1121 15:04:57.464135 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:04:57.782044 3798194 disk_info.go:125] begin check local disk info of client I1121 15:04:57.785897 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:04:57.785905 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ebf00 0xc0003ebf40] I1121 15:04:58.446028 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:04:59.450130 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:04:59.450216 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:04:59.450246 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:04:59.470730 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:05:06.413142 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:05:06.413164 3798194 memory.go:184] no items to output this cycle I1121 15:05:06.413304 3798194 cpu.go:275] no items to output this cycle E1121 15:05:16.411568 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:05:16.411588 3798194 memory.go:184] no items to output this cycle I1121 15:05:16.411841 3798194 cpu.go:275] no items to output this cycle E1121 15:05:26.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:05:26.413268 3798194 memory.go:191] Add success. I1121 15:05:26.417769 3798194 cpu.go:282] Add success. I1121 15:05:26.441267 3798194 net.go:642] Add success. I1121 15:05:26.442752 3798194 net.go:778] primary dev: ETH0 I1121 15:05:26.442768 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:05:26.442790 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:05:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:05:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:05:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:05:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:05:36.411977 3798194 memory.go:184] no items to output this cycle I1121 15:05:36.412134 3798194 cpu.go:275] no items to output this cycle E1121 15:05:46.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:05:46.411990 3798194 memory.go:184] no items to output this cycle I1121 15:05:46.412150 3798194 cpu.go:275] no items to output this cycle E1121 15:05:56.412222 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:05:56.412255 3798194 memory.go:191] Add success. W1121 15:05:56.412289 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:05:56.412301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:05:56.412304 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:05:56.416897 3798194 cpu.go:282] Add success. I1121 15:05:56.444311 3798194 event_worker.go:152] Polling the log file for events... I1121 15:05:56.445728 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:05:56.446244 3798194 net.go:642] Add success. I1121 15:05:56.447519 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795242956, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 15:05:56.447636 3798194 net.go:778] primary dev: ETH0 I1121 15:05:56.447649 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:05:56.447660 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:05:56.455509 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 15:05:57.448273 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 15:05:57.448288 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:05:57.448497 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:05:57.448570 3798194 disk_worker.go:723] disk space is not compliant W1121 15:05:57.448573 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:05:57.455853 3798194 disk_worker.go:509] system disk:vda1 I1121 15:05:57.455901 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:05:57.787149 3798194 disk_info.go:125] begin check local disk info of client I1121 15:05:57.791012 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:05:57.791022 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000359640 0xc000359680] E1121 15:05:58.448200 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 15:05:58.448216 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:05:59.449910 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:05:59.449964 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:05:59.449983 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:05:59.450124 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 15:05:59.471546 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:06:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:06:06.411976 3798194 memory.go:184] no items to output this cycle I1121 15:06:06.412131 3798194 cpu.go:275] no items to output this cycle E1121 15:06:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:06:16.411975 3798194 memory.go:184] no items to output this cycle I1121 15:06:16.412134 3798194 cpu.go:275] no items to output this cycle I1121 15:06:17.492438 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:06:17.492447 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:06:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:06:26.413271 3798194 memory.go:191] Add success. I1121 15:06:26.417451 3798194 cpu.go:282] Add success. I1121 15:06:26.442022 3798194 net.go:642] Add success. I1121 15:06:26.443544 3798194 net.go:778] primary dev: ETH0 I1121 15:06:26.443559 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:06:26.443573 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:06:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:06:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:06:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:06:36.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:06:36.413261 3798194 memory.go:184] no items to output this cycle I1121 15:06:36.413404 3798194 cpu.go:275] no items to output this cycle E1121 15:06:46.411972 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:06:46.412000 3798194 memory.go:184] no items to output this cycle I1121 15:06:46.412382 3798194 cpu.go:275] no items to output this cycle E1121 15:06:56.411929 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:06:56.411962 3798194 memory.go:191] Add success. W1121 15:06:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:06:56.412011 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:06:56.412015 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:06:56.412401 3798194 cpu.go:282] Add success. I1121 15:06:56.441890 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:06:56.442499 3798194 net.go:642] Add success. I1121 15:06:56.446927 3798194 net.go:778] primary dev: ETH0 I1121 15:06:56.446943 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:06:56.446955 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:06:57.451088 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:06:57.451287 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:06:57.451341 3798194 disk_worker.go:723] disk space is not compliant W1121 15:06:57.451345 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:06:57.465071 3798194 disk_worker.go:509] system disk:vda1 I1121 15:06:57.465131 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:06:57.791148 3798194 disk_info.go:125] begin check local disk info of client I1121 15:06:57.795009 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:06:57.795018 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000359900 0xc000359940] I1121 15:06:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:06:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:06:59.450128 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:06:59.450159 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:06:59.471703 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:07:06.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:07:06.411984 3798194 memory.go:184] no items to output this cycle I1121 15:07:06.412142 3798194 cpu.go:275] no items to output this cycle E1121 15:07:16.412847 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:07:16.412875 3798194 memory.go:184] no items to output this cycle I1121 15:07:16.413055 3798194 cpu.go:275] no items to output this cycle E1121 15:07:26.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:07:26.412896 3798194 memory.go:191] Add success. I1121 15:07:26.417338 3798194 cpu.go:282] Add success. I1121 15:07:26.441933 3798194 net.go:642] Add success. I1121 15:07:26.443743 3798194 net.go:778] primary dev: ETH0 I1121 15:07:26.443757 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:07:26.443769 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:07:29.450590 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:07:29.450674 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:07:29.450699 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:07:36.412356 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:07:36.412378 3798194 memory.go:184] no items to output this cycle I1121 15:07:36.412522 3798194 cpu.go:275] no items to output this cycle E1121 15:07:46.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:07:46.411988 3798194 memory.go:184] no items to output this cycle I1121 15:07:46.412150 3798194 cpu.go:275] no items to output this cycle E1121 15:07:56.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:07:56.412902 3798194 memory.go:191] Add success. W1121 15:07:56.412937 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:07:56.412949 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:07:56.412952 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:07:56.417445 3798194 cpu.go:282] Add success. I1121 15:07:56.441816 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:07:56.442416 3798194 net.go:642] Add success. I1121 15:07:56.443977 3798194 net.go:778] primary dev: ETH0 I1121 15:07:56.443992 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:07:56.444007 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:07:57.451740 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:07:57.451955 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:07:57.451966 3798194 disk_worker.go:723] disk space is not compliant W1121 15:07:57.451969 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:07:57.460894 3798194 disk_worker.go:509] system disk:vda1 I1121 15:07:57.460936 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:07:57.795135 3798194 disk_info.go:125] begin check local disk info of client I1121 15:07:57.799018 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:07:57.799027 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047c480 0xc00047c4c0] I1121 15:07:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:07:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:07:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:07:59.450149 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:07:59.471675 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:08:06.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:08:06.412878 3798194 memory.go:184] no items to output this cycle I1121 15:08:06.413021 3798194 cpu.go:275] no items to output this cycle E1121 15:08:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:08:16.412263 3798194 memory.go:184] no items to output this cycle I1121 15:08:16.412402 3798194 cpu.go:275] no items to output this cycle E1121 15:08:26.411445 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:08:26.411480 3798194 memory.go:191] Add success. I1121 15:08:26.415559 3798194 cpu.go:282] Add success. I1121 15:08:26.442136 3798194 net.go:642] Add success. I1121 15:08:26.449719 3798194 net.go:778] primary dev: ETH0 I1121 15:08:26.449736 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:08:26.449749 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:08:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:08:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:08:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:08:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:08:36.411980 3798194 memory.go:184] no items to output this cycle I1121 15:08:36.412132 3798194 cpu.go:275] no items to output this cycle E1121 15:08:46.413250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:08:46.413277 3798194 memory.go:184] no items to output this cycle I1121 15:08:46.413426 3798194 cpu.go:275] no items to output this cycle E1121 15:08:56.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:08:56.412270 3798194 memory.go:191] Add success. W1121 15:08:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:08:56.412317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:08:56.412321 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:08:56.416897 3798194 cpu.go:282] Add success. I1121 15:08:56.441295 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:08:56.441778 3798194 net.go:642] Add success. I1121 15:08:56.443467 3798194 net.go:778] primary dev: ETH0 I1121 15:08:56.443482 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:08:56.443495 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:08:57.456890 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:08:57.457070 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:08:57.457145 3798194 disk_worker.go:723] disk space is not compliant W1121 15:08:57.457149 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:08:57.465917 3798194 disk_worker.go:509] system disk:vda1 I1121 15:08:57.465957 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:08:57.799136 3798194 disk_info.go:125] begin check local disk info of client I1121 15:08:57.802973 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:08:57.802981 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00038b8c0 0xc00038b900] I1121 15:08:58.446391 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:08:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:08:59.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:08:59.450132 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:08:59.471650 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:09:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:09:06.411985 3798194 memory.go:184] no items to output this cycle I1121 15:09:06.412142 3798194 cpu.go:275] no items to output this cycle E1121 15:09:16.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:09:16.412262 3798194 memory.go:184] no items to output this cycle I1121 15:09:16.412405 3798194 cpu.go:275] no items to output this cycle I1121 15:09:17.492610 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:09:17.492619 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:09:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:09:26.413266 3798194 memory.go:191] Add success. I1121 15:09:26.417944 3798194 cpu.go:282] Add success. I1121 15:09:26.441485 3798194 net.go:642] Add success. I1121 15:09:26.443072 3798194 net.go:778] primary dev: ETH0 I1121 15:09:26.443089 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:09:26.443128 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:09:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:09:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:09:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:09:36.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:09:36.412871 3798194 memory.go:184] no items to output this cycle I1121 15:09:36.413012 3798194 cpu.go:275] no items to output this cycle E1121 15:09:46.411971 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:09:46.411994 3798194 memory.go:184] no items to output this cycle I1121 15:09:46.412162 3798194 cpu.go:275] no items to output this cycle E1121 15:09:56.413180 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:09:56.413215 3798194 memory.go:191] Add success. W1121 15:09:56.413248 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:09:56.413260 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:09:56.413263 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:09:56.417569 3798194 cpu.go:282] Add success. I1121 15:09:56.440937 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:09:56.441508 3798194 net.go:642] Add success. I1121 15:09:56.443026 3798194 net.go:778] primary dev: ETH0 I1121 15:09:56.443040 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:09:56.443052 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:09:57.449304 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:09:57.449507 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:09:57.449564 3798194 disk_worker.go:723] disk space is not compliant W1121 15:09:57.449567 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:09:57.451457 3798194 disk_worker.go:509] system disk:vda1 I1121 15:09:57.451500 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:09:57.804148 3798194 disk_info.go:125] begin check local disk info of client I1121 15:09:57.808024 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:09:57.808032 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000358bc0 0xc000358c00] I1121 15:09:58.446204 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:09:59.450599 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:09:59.450678 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:09:59.450703 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:09:59.471222 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:10:06.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:10:06.412878 3798194 memory.go:184] no items to output this cycle I1121 15:10:06.413030 3798194 cpu.go:275] no items to output this cycle E1121 15:10:16.413748 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:10:16.413773 3798194 memory.go:184] no items to output this cycle I1121 15:10:16.413918 3798194 cpu.go:275] no items to output this cycle E1121 15:10:26.412690 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:10:26.412732 3798194 memory.go:191] Add success. I1121 15:10:26.417224 3798194 cpu.go:282] Add success. I1121 15:10:26.441661 3798194 net.go:642] Add success. I1121 15:10:26.443302 3798194 net.go:778] primary dev: ETH0 I1121 15:10:26.443318 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:10:26.443330 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:10:29.450403 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:10:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:10:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:10:36.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:10:36.413271 3798194 memory.go:184] no items to output this cycle I1121 15:10:36.413416 3798194 cpu.go:275] no items to output this cycle E1121 15:10:46.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:10:46.412264 3798194 memory.go:184] no items to output this cycle I1121 15:10:46.412414 3798194 cpu.go:275] no items to output this cycle E1121 15:10:56.411010 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:10:56.411045 3798194 memory.go:191] Add success. W1121 15:10:56.411073 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:10:56.411085 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:10:56.411088 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:10:56.415437 3798194 cpu.go:282] Add success. I1121 15:10:56.444695 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:10:56.445320 3798194 net.go:642] Add success. I1121 15:10:56.447250 3798194 net.go:778] primary dev: ETH0 I1121 15:10:56.447267 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:10:56.447287 3798194 net.go:659] Add eni metrics success, count: 7 E1121 15:10:57.450697 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 15:10:57.450711 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:10:57.450865 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:10:57.450918 3798194 disk_worker.go:723] disk space is not compliant W1121 15:10:57.450921 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:10:57.459325 3798194 disk_worker.go:509] system disk:vda1 I1121 15:10:57.459367 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:10:57.808142 3798194 disk_info.go:125] begin check local disk info of client I1121 15:10:57.811980 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:10:57.811989 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004e1140 0xc0004e1180] E1121 15:10:58.448084 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 15:10:58.448099 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 15:10:59.448645 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 15:10:59.449722 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:10:59.449772 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:10:59.449790 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:10:59.471311 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:11:06.412251 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:11:06.412273 3798194 memory.go:184] no items to output this cycle I1121 15:11:06.412438 3798194 cpu.go:275] no items to output this cycle E1121 15:11:16.413063 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:11:16.413086 3798194 memory.go:184] no items to output this cycle I1121 15:11:16.413271 3798194 cpu.go:275] no items to output this cycle E1121 15:11:26.413254 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:11:26.413287 3798194 memory.go:191] Add success. I1121 15:11:26.418003 3798194 cpu.go:282] Add success. I1121 15:11:26.441578 3798194 net.go:642] Add success. I1121 15:11:26.443209 3798194 net.go:778] primary dev: ETH0 I1121 15:11:26.443225 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:11:26.443238 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:11:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:11:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:11:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:11:36.412865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:11:36.412888 3798194 memory.go:184] no items to output this cycle I1121 15:11:36.413033 3798194 cpu.go:275] no items to output this cycle E1121 15:11:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:11:46.413255 3798194 memory.go:184] no items to output this cycle I1121 15:11:46.413399 3798194 cpu.go:275] no items to output this cycle E1121 15:11:56.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:11:56.413266 3798194 memory.go:191] Add success. W1121 15:11:56.413300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:11:56.413314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:11:56.413317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:11:56.417995 3798194 cpu.go:282] Add success. I1121 15:11:56.441447 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:11:56.442044 3798194 net.go:642] Add success. I1121 15:11:56.443874 3798194 net.go:778] primary dev: ETH0 I1121 15:11:56.443890 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:11:56.443902 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:11:57.449175 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:11:57.449370 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:11:57.449431 3798194 disk_worker.go:723] disk space is not compliant W1121 15:11:57.449435 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:11:57.451241 3798194 disk_worker.go:509] system disk:vda1 I1121 15:11:57.451283 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:11:57.812082 3798194 disk_info.go:125] begin check local disk info of client I1121 15:11:57.815971 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:11:57.815980 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00038c040 0xc00038c080] I1121 15:11:58.446355 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:11:59.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:11:59.450466 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:11:59.450491 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:11:59.471029 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:12:06.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:12:06.413249 3798194 memory.go:184] no items to output this cycle I1121 15:12:06.413396 3798194 cpu.go:275] no items to output this cycle E1121 15:12:16.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:12:16.413262 3798194 memory.go:184] no items to output this cycle I1121 15:12:16.413534 3798194 cpu.go:275] no items to output this cycle I1121 15:12:17.493448 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:12:17.493458 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:12:26.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:12:26.413278 3798194 memory.go:191] Add success. I1121 15:12:26.417486 3798194 cpu.go:282] Add success. I1121 15:12:26.442028 3798194 net.go:642] Add success. I1121 15:12:26.443619 3798194 net.go:778] primary dev: ETH0 I1121 15:12:26.443635 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:12:26.443649 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:12:29.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:12:29.450466 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:12:29.450495 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:12:36.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:12:36.413252 3798194 memory.go:184] no items to output this cycle I1121 15:12:36.413398 3798194 cpu.go:275] no items to output this cycle E1121 15:12:46.412434 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:12:46.412458 3798194 memory.go:184] no items to output this cycle I1121 15:12:46.412607 3798194 cpu.go:275] no items to output this cycle E1121 15:12:56.413224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:12:56.413257 3798194 memory.go:191] Add success. W1121 15:12:56.413300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:12:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:12:56.413315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:12:56.417818 3798194 cpu.go:282] Add success. I1121 15:12:56.441221 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:12:56.441785 3798194 net.go:642] Add success. I1121 15:12:56.443447 3798194 net.go:778] primary dev: ETH0 I1121 15:12:56.443461 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:12:56.443476 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:12:57.450653 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:12:57.450859 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:12:57.450871 3798194 disk_worker.go:723] disk space is not compliant W1121 15:12:57.450874 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:12:57.459337 3798194 disk_worker.go:509] system disk:vda1 I1121 15:12:57.459378 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:12:57.817144 3798194 disk_info.go:125] begin check local disk info of client I1121 15:12:57.820974 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:12:57.820983 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000335bc0 0xc000335c00] I1121 15:12:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:12:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:12:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:12:59.450148 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:12:59.471673 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:13:06.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:13:06.412266 3798194 memory.go:184] no items to output this cycle I1121 15:13:06.412410 3798194 cpu.go:275] no items to output this cycle E1121 15:13:16.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:13:16.411987 3798194 memory.go:184] no items to output this cycle I1121 15:13:16.412153 3798194 cpu.go:275] no items to output this cycle E1121 15:13:26.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:13:26.411961 3798194 memory.go:191] Add success. I1121 15:13:26.412321 3798194 cpu.go:282] Add success. I1121 15:13:26.441990 3798194 net.go:642] Add success. I1121 15:13:26.446382 3798194 net.go:778] primary dev: ETH0 I1121 15:13:26.446398 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:13:26.446413 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:13:29.449944 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:13:29.450015 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:13:29.450040 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:13:36.412817 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:13:36.412844 3798194 memory.go:184] no items to output this cycle I1121 15:13:36.412998 3798194 cpu.go:275] no items to output this cycle E1121 15:13:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:13:46.412257 3798194 memory.go:184] no items to output this cycle I1121 15:13:46.412413 3798194 cpu.go:275] no items to output this cycle E1121 15:13:56.412891 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:13:56.412929 3798194 memory.go:191] Add success. W1121 15:13:56.412962 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:13:56.412973 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:13:56.412977 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:13:56.420211 3798194 cpu.go:282] Add success. I1121 15:13:56.441830 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:13:56.442330 3798194 net.go:642] Add success. I1121 15:13:56.443999 3798194 net.go:778] primary dev: ETH0 I1121 15:13:56.444014 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:13:56.444026 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:13:57.451426 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:13:57.451629 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:13:57.451640 3798194 disk_worker.go:723] disk space is not compliant W1121 15:13:57.451643 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:13:57.459980 3798194 disk_worker.go:509] system disk:vda1 I1121 15:13:57.460020 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:13:57.821086 3798194 disk_info.go:125] begin check local disk info of client I1121 15:13:57.824935 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:13:57.824944 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e1500 0xc0002e1540] I1121 15:13:58.446389 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:13:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:13:59.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:13:59.450498 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:13:59.471025 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:14:06.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:14:06.411985 3798194 memory.go:184] no items to output this cycle I1121 15:14:06.412147 3798194 cpu.go:275] no items to output this cycle E1121 15:14:16.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:14:16.413257 3798194 memory.go:184] no items to output this cycle I1121 15:14:16.413407 3798194 cpu.go:275] no items to output this cycle E1121 15:14:26.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:14:26.412264 3798194 memory.go:191] Add success. I1121 15:14:26.416652 3798194 cpu.go:282] Add success. I1121 15:14:26.441450 3798194 net.go:642] Add success. I1121 15:14:26.442994 3798194 net.go:778] primary dev: ETH0 I1121 15:14:26.443009 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:14:26.443022 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:14:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:14:29.450088 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:14:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:14:36.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:14:36.413262 3798194 memory.go:184] no items to output this cycle I1121 15:14:36.413414 3798194 cpu.go:275] no items to output this cycle E1121 15:14:46.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:14:46.413260 3798194 memory.go:184] no items to output this cycle I1121 15:14:46.413415 3798194 cpu.go:275] no items to output this cycle E1121 15:14:56.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:14:56.413275 3798194 memory.go:191] Add success. W1121 15:14:56.413310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:14:56.413321 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:14:56.413324 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:14:56.418142 3798194 cpu.go:282] Add success. I1121 15:14:56.441416 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:14:56.441887 3798194 net.go:642] Add success. I1121 15:14:56.443344 3798194 net.go:778] primary dev: ETH0 I1121 15:14:56.443358 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:14:56.443372 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:14:57.446442 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:14:57.446709 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:14:57.446723 3798194 disk_worker.go:723] disk space is not compliant W1121 15:14:57.446726 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:14:57.449520 3798194 disk_worker.go:509] system disk:vda1 I1121 15:14:57.449559 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:14:57.826121 3798194 disk_info.go:125] begin check local disk info of client I1121 15:14:57.830051 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:14:57.830059 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004f22c0 0xc0004f2300] I1121 15:14:58.446355 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:14:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:14:59.450465 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:14:59.450490 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:14:59.470992 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:15:06.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:15:06.412264 3798194 memory.go:184] no items to output this cycle I1121 15:15:06.412407 3798194 cpu.go:275] no items to output this cycle E1121 15:15:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:15:16.413259 3798194 memory.go:184] no items to output this cycle I1121 15:15:16.413402 3798194 cpu.go:275] no items to output this cycle I1121 15:15:17.493623 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:15:17.493631 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:15:26.411309 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:15:26.411343 3798194 memory.go:191] Add success. I1121 15:15:26.411695 3798194 cpu.go:282] Add success. I1121 15:15:26.441293 3798194 net.go:642] Add success. I1121 15:15:26.446225 3798194 net.go:778] primary dev: ETH0 I1121 15:15:26.446241 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:15:26.446255 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:15:29.450007 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:15:29.450086 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:15:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:15:36.412350 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:15:36.412373 3798194 memory.go:184] no items to output this cycle I1121 15:15:36.412516 3798194 cpu.go:275] no items to output this cycle E1121 15:15:46.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:15:46.413260 3798194 memory.go:184] no items to output this cycle I1121 15:15:46.413406 3798194 cpu.go:275] no items to output this cycle E1121 15:15:56.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:15:56.412260 3798194 memory.go:191] Add success. W1121 15:15:56.412291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:15:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:15:56.412305 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:15:56.416926 3798194 cpu.go:282] Add success. I1121 15:15:56.441515 3798194 event_worker.go:152] Polling the log file for events... I1121 15:15:56.444684 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:15:56.445175 3798194 net.go:642] Add success. I1121 15:15:56.446875 3798194 net.go:778] primary dev: ETH0 I1121 15:15:56.446889 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:15:56.446901 3798194 net.go:659] Add eni metrics success, count: 7 E1121 15:15:57.450607 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 15:15:57.450620 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:15:57.450775 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:15:57.450826 3798194 disk_worker.go:723] disk space is not compliant W1121 15:15:57.450829 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:15:57.459207 3798194 disk_worker.go:509] system disk:vda1 I1121 15:15:57.459249 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:15:57.831137 3798194 disk_info.go:125] begin check local disk info of client I1121 15:15:57.834990 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:15:57.835000 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b36c0 0xc0003b3700] E1121 15:15:58.448080 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 15:15:58.448094 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 15:15:59.448608 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 15:15:59.449684 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:15:59.449733 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:15:59.449752 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:15:59.471260 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:16:06.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:16:06.411971 3798194 memory.go:184] no items to output this cycle I1121 15:16:06.412136 3798194 cpu.go:275] no items to output this cycle E1121 15:16:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:16:16.412259 3798194 memory.go:184] no items to output this cycle I1121 15:16:16.412418 3798194 cpu.go:275] no items to output this cycle E1121 15:16:26.413224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:16:26.413256 3798194 memory.go:191] Add success. I1121 15:16:26.417642 3798194 cpu.go:282] Add success. I1121 15:16:26.441253 3798194 net.go:642] Add success. I1121 15:16:26.442810 3798194 net.go:778] primary dev: ETH0 I1121 15:16:26.442826 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:16:26.442949 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:16:29.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:16:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:16:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:16:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:16:36.413264 3798194 memory.go:184] no items to output this cycle I1121 15:16:36.413408 3798194 cpu.go:275] no items to output this cycle E1121 15:16:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:16:46.411982 3798194 memory.go:184] no items to output this cycle I1121 15:16:46.412140 3798194 cpu.go:275] no items to output this cycle E1121 15:16:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:16:56.411952 3798194 memory.go:191] Add success. W1121 15:16:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:16:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:16:56.412000 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:16:56.412357 3798194 cpu.go:282] Add success. I1121 15:16:56.441814 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:16:56.442536 3798194 net.go:642] Add success. I1121 15:16:56.446818 3798194 net.go:778] primary dev: ETH0 I1121 15:16:56.446834 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:16:56.446847 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:16:57.451346 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:16:57.451513 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:16:57.451567 3798194 disk_worker.go:723] disk space is not compliant W1121 15:16:57.451570 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:16:57.459976 3798194 disk_worker.go:509] system disk:vda1 I1121 15:16:57.460019 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:16:57.836141 3798194 disk_info.go:125] begin check local disk info of client I1121 15:16:57.839887 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:16:57.839898 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ff400 0xc0003ff440] I1121 15:16:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:16:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:16:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:16:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:16:59.471646 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:17:06.412767 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:17:06.412794 3798194 memory.go:184] no items to output this cycle I1121 15:17:06.412953 3798194 cpu.go:275] no items to output this cycle E1121 15:17:16.412333 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:17:16.412354 3798194 memory.go:184] no items to output this cycle I1121 15:17:16.412502 3798194 cpu.go:275] no items to output this cycle E1121 15:17:26.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:17:26.412275 3798194 memory.go:191] Add success. I1121 15:17:26.416541 3798194 cpu.go:282] Add success. I1121 15:17:26.442156 3798194 net.go:642] Add success. I1121 15:17:26.443753 3798194 net.go:778] primary dev: ETH0 I1121 15:17:26.443768 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:17:26.443779 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:17:29.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:17:29.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:17:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:17:36.412997 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:17:36.413019 3798194 memory.go:184] no items to output this cycle I1121 15:17:36.413193 3798194 cpu.go:275] no items to output this cycle E1121 15:17:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:17:46.411971 3798194 memory.go:184] no items to output this cycle I1121 15:17:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 15:17:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:17:56.413264 3798194 memory.go:191] Add success. W1121 15:17:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:17:56.413309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:17:56.413312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:17:56.417643 3798194 cpu.go:282] Add success. I1121 15:17:56.440967 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:17:56.441569 3798194 net.go:642] Add success. I1121 15:17:56.443390 3798194 net.go:778] primary dev: ETH0 I1121 15:17:56.443404 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:17:56.443416 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:17:57.451131 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:17:57.451298 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:17:57.451354 3798194 disk_worker.go:723] disk space is not compliant W1121 15:17:57.451357 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:17:57.459721 3798194 disk_worker.go:509] system disk:vda1 I1121 15:17:57.459760 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:17:57.840000 3798194 disk_info.go:125] begin check local disk info of client I1121 15:17:57.843876 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:17:57.843885 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004c1e80 0xc0004c1ec0] I1121 15:17:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:17:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:17:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:17:59.450125 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:17:59.471664 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:18:06.412836 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:18:06.412865 3798194 memory.go:184] no items to output this cycle I1121 15:18:06.413009 3798194 cpu.go:275] no items to output this cycle E1121 15:18:16.412769 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:18:16.412793 3798194 memory.go:184] no items to output this cycle I1121 15:18:16.412938 3798194 cpu.go:275] no items to output this cycle I1121 15:18:17.494275 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:18:17.494284 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:18:26.411317 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:18:26.411353 3798194 memory.go:191] Add success. I1121 15:18:26.415604 3798194 cpu.go:282] Add success. I1121 15:18:26.442185 3798194 net.go:642] Add success. I1121 15:18:26.443759 3798194 net.go:778] primary dev: ETH0 I1121 15:18:26.443774 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:18:26.443789 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:18:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:18:29.450465 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:18:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:18:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:18:36.411979 3798194 memory.go:184] no items to output this cycle I1121 15:18:36.412149 3798194 cpu.go:275] no items to output this cycle E1121 15:18:46.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:18:46.413257 3798194 memory.go:184] no items to output this cycle I1121 15:18:46.413409 3798194 cpu.go:275] no items to output this cycle E1121 15:18:56.411932 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:18:56.411964 3798194 memory.go:191] Add success. W1121 15:18:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:18:56.412009 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:18:56.412012 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:18:56.412349 3798194 cpu.go:282] Add success. I1121 15:18:56.441818 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:18:56.442343 3798194 net.go:642] Add success. I1121 15:18:56.446342 3798194 net.go:778] primary dev: ETH0 I1121 15:18:56.446357 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:18:56.446371 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:18:57.451011 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:18:57.451251 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:18:57.451262 3798194 disk_worker.go:723] disk space is not compliant W1121 15:18:57.451266 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:18:57.460136 3798194 disk_worker.go:509] system disk:vda1 I1121 15:18:57.460180 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:18:57.845136 3798194 disk_info.go:125] begin check local disk info of client I1121 15:18:57.848945 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:18:57.848954 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041f0c0 0xc00041f100] I1121 15:18:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:18:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:18:59.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:18:59.450148 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:18:59.471687 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:19:06.412091 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:19:06.412131 3798194 memory.go:184] no items to output this cycle I1121 15:19:06.412279 3798194 cpu.go:275] no items to output this cycle E1121 15:19:16.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:19:16.413264 3798194 memory.go:184] no items to output this cycle I1121 15:19:16.413407 3798194 cpu.go:275] no items to output this cycle E1121 15:19:26.412045 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:19:26.412166 3798194 memory.go:191] Add success. I1121 15:19:26.412818 3798194 cpu.go:282] Add success. I1121 15:19:26.441311 3798194 net.go:642] Add success. I1121 15:19:26.445505 3798194 net.go:778] primary dev: ETH0 I1121 15:19:26.445521 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:19:26.445533 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:19:29.449975 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:19:29.450056 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:19:29.450085 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:19:36.412869 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:19:36.412890 3798194 memory.go:184] no items to output this cycle I1121 15:19:36.413045 3798194 cpu.go:275] no items to output this cycle E1121 15:19:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:19:46.412256 3798194 memory.go:184] no items to output this cycle I1121 15:19:46.412422 3798194 cpu.go:275] no items to output this cycle E1121 15:19:56.411930 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:19:56.411967 3798194 memory.go:191] Add success. W1121 15:19:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:19:56.412011 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:19:56.412014 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:19:56.412394 3798194 cpu.go:282] Add success. I1121 15:19:56.441867 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:19:56.442402 3798194 net.go:642] Add success. I1121 15:19:56.446464 3798194 net.go:778] primary dev: ETH0 I1121 15:19:56.446480 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:19:56.446491 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:19:57.450806 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:19:57.450970 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:19:57.451026 3798194 disk_worker.go:723] disk space is not compliant W1121 15:19:57.451029 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:19:57.459367 3798194 disk_worker.go:509] system disk:vda1 I1121 15:19:57.459410 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:19:57.849057 3798194 disk_info.go:125] begin check local disk info of client I1121 15:19:57.852865 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:19:57.852874 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00015cb40 0xc00015cb80] I1121 15:19:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:19:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:19:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:19:59.450153 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:19:59.471667 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:20:06.412248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:20:06.412274 3798194 memory.go:184] no items to output this cycle I1121 15:20:06.412417 3798194 cpu.go:275] no items to output this cycle E1121 15:20:16.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:20:16.413271 3798194 memory.go:184] no items to output this cycle I1121 15:20:16.413419 3798194 cpu.go:275] no items to output this cycle E1121 15:20:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:20:26.413267 3798194 memory.go:191] Add success. I1121 15:20:26.418192 3798194 cpu.go:282] Add success. I1121 15:20:26.441687 3798194 net.go:642] Add success. I1121 15:20:26.443312 3798194 net.go:778] primary dev: ETH0 I1121 15:20:26.443327 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:20:26.443341 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:20:29.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:20:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:20:29.450501 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:20:36.412539 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:20:36.412563 3798194 memory.go:184] no items to output this cycle I1121 15:20:36.412711 3798194 cpu.go:275] no items to output this cycle E1121 15:20:46.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:20:46.412255 3798194 memory.go:184] no items to output this cycle I1121 15:20:46.412413 3798194 cpu.go:275] no items to output this cycle E1121 15:20:56.412372 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:20:56.412410 3798194 memory.go:191] Add success. W1121 15:20:56.412438 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:20:56.412449 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:20:56.412453 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:20:56.417168 3798194 cpu.go:282] Add success. I1121 15:20:56.444423 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:20:56.444919 3798194 net.go:642] Add success. I1121 15:20:56.446383 3798194 net.go:778] primary dev: ETH0 I1121 15:20:56.446398 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:20:56.446410 3798194 net.go:659] Add eni metrics success, count: 7 E1121 15:20:57.450494 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 15:20:57.450508 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:20:57.450656 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:20:57.450720 3798194 disk_worker.go:723] disk space is not compliant W1121 15:20:57.450723 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:20:57.459069 3798194 disk_worker.go:509] system disk:vda1 I1121 15:20:57.459124 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:20:57.852978 3798194 disk_info.go:125] begin check local disk info of client I1121 15:20:57.856833 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:20:57.856842 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0500 0xc0002e0540] E1121 15:20:58.448451 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 15:20:58.448464 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 15:20:59.449021 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 15:20:59.450099 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:20:59.450158 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:20:59.450176 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:20:59.471692 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:21:06.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:21:06.413271 3798194 memory.go:184] no items to output this cycle I1121 15:21:06.413418 3798194 cpu.go:275] no items to output this cycle E1121 15:21:16.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:21:16.411989 3798194 memory.go:184] no items to output this cycle I1121 15:21:16.412146 3798194 cpu.go:275] no items to output this cycle I1121 15:21:17.495428 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:21:17.495437 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:21:26.411294 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:21:26.411328 3798194 memory.go:191] Add success. I1121 15:21:26.411669 3798194 cpu.go:282] Add success. I1121 15:21:26.441311 3798194 net.go:642] Add success. I1121 15:21:26.445469 3798194 net.go:778] primary dev: ETH0 I1121 15:21:26.445487 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:21:26.445499 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:21:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:21:29.450124 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:21:29.450156 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:21:36.412335 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:21:36.412358 3798194 memory.go:184] no items to output this cycle I1121 15:21:36.412508 3798194 cpu.go:275] no items to output this cycle E1121 15:21:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:21:46.411982 3798194 memory.go:184] no items to output this cycle I1121 15:21:46.412144 3798194 cpu.go:275] no items to output this cycle E1121 15:21:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:21:56.411956 3798194 memory.go:191] Add success. W1121 15:21:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:21:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:21:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:21:56.412360 3798194 cpu.go:282] Add success. I1121 15:21:56.441824 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:21:56.442360 3798194 net.go:642] Add success. I1121 15:21:56.446342 3798194 net.go:778] primary dev: ETH0 I1121 15:21:56.446358 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:21:56.446374 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:21:57.450612 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:21:57.450821 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:21:57.450833 3798194 disk_worker.go:723] disk space is not compliant W1121 15:21:57.450836 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:21:57.459927 3798194 disk_worker.go:509] system disk:vda1 I1121 15:21:57.459964 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:21:57.856945 3798194 disk_info.go:125] begin check local disk info of client I1121 15:21:57.860767 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:21:57.860777 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000475680 0xc000475780] I1121 15:21:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:21:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:21:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:21:59.450144 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:21:59.470688 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:22:06.413183 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:22:06.413205 3798194 memory.go:184] no items to output this cycle I1121 15:22:06.413352 3798194 cpu.go:275] no items to output this cycle E1121 15:22:16.412254 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:22:16.412277 3798194 memory.go:184] no items to output this cycle I1121 15:22:16.412426 3798194 cpu.go:275] no items to output this cycle E1121 15:22:26.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:22:26.411947 3798194 memory.go:191] Add success. I1121 15:22:26.412278 3798194 cpu.go:282] Add success. I1121 15:22:26.441748 3798194 net.go:642] Add success. I1121 15:22:26.445525 3798194 net.go:778] primary dev: ETH0 I1121 15:22:26.445541 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:22:26.445554 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:22:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:22:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:22:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:22:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:22:36.412254 3798194 memory.go:184] no items to output this cycle I1121 15:22:36.412395 3798194 cpu.go:275] no items to output this cycle E1121 15:22:46.412611 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:22:46.412633 3798194 memory.go:184] no items to output this cycle I1121 15:22:46.412790 3798194 cpu.go:275] no items to output this cycle E1121 15:22:56.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:22:56.413272 3798194 memory.go:191] Add success. W1121 15:22:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:22:56.413318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:22:56.413321 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:22:56.417751 3798194 cpu.go:282] Add success. I1121 15:22:56.441147 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:22:56.441639 3798194 net.go:642] Add success. I1121 15:22:56.443186 3798194 net.go:778] primary dev: ETH0 I1121 15:22:56.443201 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:22:56.443216 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:22:57.454566 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:22:57.454732 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:22:57.454788 3798194 disk_worker.go:723] disk space is not compliant W1121 15:22:57.454792 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:22:57.475456 3798194 disk_worker.go:509] system disk:vda1 I1121 15:22:57.475500 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:22:57.860869 3798194 disk_info.go:125] begin check local disk info of client I1121 15:22:57.864810 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:22:57.864818 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003cc1c0 0xc0003cc200] I1121 15:22:58.446357 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:22:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:22:59.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:22:59.450510 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:22:59.471034 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:23:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:23:06.411979 3798194 memory.go:184] no items to output this cycle I1121 15:23:06.412148 3798194 cpu.go:275] no items to output this cycle E1121 15:23:16.412336 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:23:16.412358 3798194 memory.go:184] no items to output this cycle I1121 15:23:16.412515 3798194 cpu.go:275] no items to output this cycle E1121 15:23:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:23:26.413266 3798194 memory.go:191] Add success. I1121 15:23:26.417498 3798194 cpu.go:282] Add success. I1121 15:23:26.441884 3798194 net.go:642] Add success. I1121 15:23:26.443496 3798194 net.go:778] primary dev: ETH0 I1121 15:23:26.443512 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:23:26.443526 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:23:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:23:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:23:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:23:36.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:23:36.412886 3798194 memory.go:184] no items to output this cycle I1121 15:23:36.413035 3798194 cpu.go:275] no items to output this cycle E1121 15:23:46.413081 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:23:46.413103 3798194 memory.go:184] no items to output this cycle I1121 15:23:46.413263 3798194 cpu.go:275] no items to output this cycle E1121 15:23:56.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:23:56.411958 3798194 memory.go:191] Add success. W1121 15:23:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:23:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:23:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:23:56.412358 3798194 cpu.go:282] Add success. I1121 15:23:56.441823 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:23:56.442350 3798194 net.go:642] Add success. I1121 15:23:56.447067 3798194 net.go:778] primary dev: ETH0 I1121 15:23:56.447081 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:23:56.447095 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:23:57.450645 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:23:57.450841 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:23:57.450855 3798194 disk_worker.go:723] disk space is not compliant W1121 15:23:57.450858 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:23:57.459239 3798194 disk_worker.go:509] system disk:vda1 I1121 15:23:57.459281 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:23:57.866146 3798194 disk_info.go:125] begin check local disk info of client I1121 15:23:57.869998 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:23:57.870007 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003cc3c0 0xc0003cc400] I1121 15:23:58.446011 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:23:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:23:59.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:23:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:23:59.471640 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:24:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:24:06.412259 3798194 memory.go:184] no items to output this cycle I1121 15:24:06.412405 3798194 cpu.go:275] no items to output this cycle E1121 15:24:16.413407 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:24:16.413429 3798194 memory.go:184] no items to output this cycle I1121 15:24:16.413574 3798194 cpu.go:275] no items to output this cycle I1121 15:24:17.496269 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:24:17.496278 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:24:26.411292 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:24:26.411329 3798194 memory.go:191] Add success. I1121 15:24:26.411658 3798194 cpu.go:282] Add success. I1121 15:24:26.442284 3798194 net.go:642] Add success. I1121 15:24:26.446330 3798194 net.go:778] primary dev: ETH0 I1121 15:24:26.446345 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:24:26.446359 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:24:29.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:24:29.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:24:29.450149 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:24:36.412292 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:24:36.412314 3798194 memory.go:184] no items to output this cycle I1121 15:24:36.412458 3798194 cpu.go:275] no items to output this cycle E1121 15:24:46.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:24:46.413264 3798194 memory.go:184] no items to output this cycle I1121 15:24:46.413409 3798194 cpu.go:275] no items to output this cycle E1121 15:24:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:24:56.411962 3798194 memory.go:191] Add success. W1121 15:24:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:24:56.412008 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:24:56.412011 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:24:56.412371 3798194 cpu.go:282] Add success. I1121 15:24:56.441792 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:24:56.442294 3798194 net.go:642] Add success. I1121 15:24:56.446347 3798194 net.go:778] primary dev: ETH0 I1121 15:24:56.446363 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:24:56.446377 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:24:57.450721 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:24:57.450933 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:24:57.450944 3798194 disk_worker.go:723] disk space is not compliant W1121 15:24:57.450947 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:24:57.459345 3798194 disk_worker.go:509] system disk:vda1 I1121 15:24:57.459385 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:24:57.871143 3798194 disk_info.go:125] begin check local disk info of client I1121 15:24:57.875021 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:24:57.875029 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0140 0xc0002e0180] I1121 15:24:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:24:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:24:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:24:59.450126 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:24:59.471652 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:25:06.412779 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:25:06.412801 3798194 memory.go:184] no items to output this cycle I1121 15:25:06.412955 3798194 cpu.go:275] no items to output this cycle E1121 15:25:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:25:16.411982 3798194 memory.go:184] no items to output this cycle I1121 15:25:16.412142 3798194 cpu.go:275] no items to output this cycle E1121 15:25:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:25:26.411952 3798194 memory.go:191] Add success. I1121 15:25:26.412309 3798194 cpu.go:282] Add success. I1121 15:25:26.441918 3798194 net.go:642] Add success. I1121 15:25:26.446450 3798194 net.go:778] primary dev: ETH0 I1121 15:25:26.446465 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:25:26.446478 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:25:29.450193 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:25:29.450276 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:25:29.450307 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:25:36.413167 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:25:36.413190 3798194 memory.go:184] no items to output this cycle I1121 15:25:36.413345 3798194 cpu.go:275] no items to output this cycle E1121 15:25:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:25:46.411982 3798194 memory.go:184] no items to output this cycle I1121 15:25:46.412142 3798194 cpu.go:275] no items to output this cycle E1121 15:25:56.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:25:56.412890 3798194 memory.go:191] Add success. W1121 15:25:56.412924 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:25:56.412936 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:25:56.412941 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:25:56.417413 3798194 cpu.go:282] Add success. I1121 15:25:56.441548 3798194 event_worker.go:152] Polling the log file for events... I1121 15:25:56.444687 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:25:56.445196 3798194 net.go:642] Add success. I1121 15:25:56.446728 3798194 net.go:778] primary dev: ETH0 I1121 15:25:56.446743 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:25:56.446756 3798194 net.go:659] Add eni metrics success, count: 7 E1121 15:25:57.451285 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 15:25:57.451299 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:25:57.451500 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:25:57.451510 3798194 disk_worker.go:723] disk space is not compliant W1121 15:25:57.451513 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:25:57.460651 3798194 disk_worker.go:509] system disk:vda1 I1121 15:25:57.460689 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:25:57.875144 3798194 disk_info.go:125] begin check local disk info of client I1121 15:25:57.879057 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:25:57.879066 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000417740 0xc000417780] E1121 15:25:58.448486 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 15:25:58.448501 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 15:25:59.449215 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 15:25:59.450290 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:25:59.450344 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:25:59.450362 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:25:59.470902 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:26:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:26:06.411981 3798194 memory.go:184] no items to output this cycle I1121 15:26:06.412138 3798194 cpu.go:275] no items to output this cycle E1121 15:26:16.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:26:16.412876 3798194 memory.go:184] no items to output this cycle I1121 15:26:16.413030 3798194 cpu.go:275] no items to output this cycle E1121 15:26:26.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:26:26.411950 3798194 memory.go:191] Add success. I1121 15:26:26.412294 3798194 cpu.go:282] Add success. I1121 15:26:26.441741 3798194 net.go:642] Add success. I1121 15:26:26.445203 3798194 net.go:778] primary dev: ETH0 I1121 15:26:26.445219 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:26:26.445232 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:26:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:26:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:26:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:26:36.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:26:36.413260 3798194 memory.go:184] no items to output this cycle I1121 15:26:36.413399 3798194 cpu.go:275] no items to output this cycle E1121 15:26:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:26:46.411984 3798194 memory.go:184] no items to output this cycle I1121 15:26:46.412143 3798194 cpu.go:275] no items to output this cycle E1121 15:26:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:26:56.411953 3798194 memory.go:191] Add success. W1121 15:26:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:26:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:26:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:26:56.412353 3798194 cpu.go:282] Add success. I1121 15:26:56.441804 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:26:56.442394 3798194 net.go:642] Add success. I1121 15:26:56.446217 3798194 net.go:778] primary dev: ETH0 I1121 15:26:56.446232 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:26:56.446243 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:26:57.450622 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:26:57.450785 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:26:57.450837 3798194 disk_worker.go:723] disk space is not compliant W1121 15:26:57.450840 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:26:57.459195 3798194 disk_worker.go:509] system disk:vda1 I1121 15:26:57.459239 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:26:57.880133 3798194 disk_info.go:125] begin check local disk info of client I1121 15:26:57.883936 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:26:57.883946 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003cc2c0 0xc0003cc300] I1121 15:26:58.445996 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:26:59.450098 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:26:59.450188 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:26:59.450214 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:26:59.470709 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:27:06.413362 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:27:06.413384 3798194 memory.go:184] no items to output this cycle I1121 15:27:06.413525 3798194 cpu.go:275] no items to output this cycle E1121 15:27:16.412342 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:27:16.412365 3798194 memory.go:184] no items to output this cycle I1121 15:27:16.412526 3798194 cpu.go:275] no items to output this cycle I1121 15:27:17.496434 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:27:17.496442 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:27:26.411324 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:27:26.411357 3798194 memory.go:191] Add success. I1121 15:27:26.411689 3798194 cpu.go:282] Add success. I1121 15:27:26.441632 3798194 net.go:642] Add success. I1121 15:27:26.446630 3798194 net.go:778] primary dev: ETH0 I1121 15:27:26.446647 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:27:26.446659 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:27:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:27:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:27:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:27:36.412813 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:27:36.412835 3798194 memory.go:184] no items to output this cycle I1121 15:27:36.412976 3798194 cpu.go:275] no items to output this cycle E1121 15:27:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:27:46.411985 3798194 memory.go:184] no items to output this cycle I1121 15:27:46.412162 3798194 cpu.go:275] no items to output this cycle E1121 15:27:56.411936 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:27:56.411981 3798194 memory.go:191] Add success. W1121 15:27:56.412019 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:27:56.412031 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:27:56.412035 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:27:56.412395 3798194 cpu.go:282] Add success. I1121 15:27:56.441855 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:27:56.442466 3798194 net.go:642] Add success. I1121 15:27:56.446546 3798194 net.go:778] primary dev: ETH0 I1121 15:27:56.446561 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:27:56.446574 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:27:57.450706 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:27:57.450900 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:27:57.450911 3798194 disk_worker.go:723] disk space is not compliant W1121 15:27:57.450914 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:27:57.459449 3798194 disk_worker.go:509] system disk:vda1 I1121 15:27:57.459498 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:27:57.885146 3798194 disk_info.go:125] begin check local disk info of client I1121 15:27:57.888980 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:27:57.888990 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003cc480 0xc0003cc4c0] I1121 15:27:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:27:59.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:27:59.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:27:59.450513 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:27:59.471055 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:28:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:28:06.412256 3798194 memory.go:184] no items to output this cycle I1121 15:28:06.412400 3798194 cpu.go:275] no items to output this cycle E1121 15:28:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:28:16.411975 3798194 memory.go:184] no items to output this cycle I1121 15:28:16.412134 3798194 cpu.go:275] no items to output this cycle E1121 15:28:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:28:26.413268 3798194 memory.go:191] Add success. I1121 15:28:26.417390 3798194 cpu.go:282] Add success. I1121 15:28:26.441952 3798194 net.go:642] Add success. I1121 15:28:26.443274 3798194 net.go:778] primary dev: ETH0 I1121 15:28:26.443289 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:28:26.443304 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:28:29.450203 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:28:29.450281 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:28:29.450305 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:28:36.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:28:36.413258 3798194 memory.go:184] no items to output this cycle I1121 15:28:36.413398 3798194 cpu.go:275] no items to output this cycle E1121 15:28:46.412948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:28:46.412971 3798194 memory.go:184] no items to output this cycle I1121 15:28:46.413141 3798194 cpu.go:275] no items to output this cycle E1121 15:28:56.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:28:56.412896 3798194 memory.go:191] Add success. W1121 15:28:56.412930 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:28:56.412942 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:28:56.412945 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:28:56.416981 3798194 cpu.go:282] Add success. I1121 15:28:56.441285 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:28:56.441862 3798194 net.go:642] Add success. I1121 15:28:56.443323 3798194 net.go:778] primary dev: ETH0 I1121 15:28:56.443338 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:28:56.443350 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:28:57.445646 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:28:57.445868 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:28:57.445939 3798194 disk_worker.go:723] disk space is not compliant W1121 15:28:57.445943 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:28:57.450967 3798194 disk_worker.go:509] system disk:vda1 I1121 15:28:57.451010 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:28:57.890146 3798194 disk_info.go:125] begin check local disk info of client I1121 15:28:57.894025 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:28:57.894034 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ed80 0xc00030edc0] I1121 15:28:58.445986 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:28:59.450504 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:28:59.450577 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:28:59.450606 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:28:59.471131 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:29:06.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:29:06.411986 3798194 memory.go:184] no items to output this cycle I1121 15:29:06.412142 3798194 cpu.go:275] no items to output this cycle E1121 15:29:16.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:29:16.413256 3798194 memory.go:184] no items to output this cycle I1121 15:29:16.413405 3798194 cpu.go:275] no items to output this cycle E1121 15:29:26.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:29:26.413261 3798194 memory.go:191] Add success. I1121 15:29:26.417434 3798194 cpu.go:282] Add success. I1121 15:29:26.441975 3798194 net.go:642] Add success. I1121 15:29:26.443350 3798194 net.go:778] primary dev: ETH0 I1121 15:29:26.443364 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:29:26.443376 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:29:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:29:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:29:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:29:36.412170 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:29:36.412191 3798194 memory.go:184] no items to output this cycle I1121 15:29:36.412335 3798194 cpu.go:275] no items to output this cycle E1121 15:29:46.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:29:46.413266 3798194 memory.go:184] no items to output this cycle I1121 15:29:46.413416 3798194 cpu.go:275] no items to output this cycle E1121 15:29:56.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:29:56.412890 3798194 memory.go:191] Add success. W1121 15:29:56.412918 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:29:56.412930 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:29:56.412933 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:29:56.417248 3798194 cpu.go:282] Add success. I1121 15:29:56.441652 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:29:56.442185 3798194 net.go:642] Add success. I1121 15:29:56.443823 3798194 net.go:778] primary dev: ETH0 I1121 15:29:56.443837 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:29:56.443850 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:29:57.451169 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:29:57.451370 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:29:57.451382 3798194 disk_worker.go:723] disk space is not compliant W1121 15:29:57.451385 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:29:57.459901 3798194 disk_worker.go:509] system disk:vda1 I1121 15:29:57.459943 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:29:57.894136 3798194 disk_info.go:125] begin check local disk info of client I1121 15:29:57.897992 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:29:57.898001 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000493140 0xc000493180] I1121 15:29:58.446383 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:29:59.450485 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:29:59.450554 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:29:59.450581 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:29:59.471126 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:30:06.412819 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:30:06.412841 3798194 memory.go:184] no items to output this cycle I1121 15:30:06.413259 3798194 cpu.go:275] no items to output this cycle E1121 15:30:16.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:30:16.411973 3798194 memory.go:184] no items to output this cycle I1121 15:30:16.412131 3798194 cpu.go:275] no items to output this cycle I1121 15:30:17.497408 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:30:17.497417 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:30:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:30:26.412268 3798194 memory.go:191] Add success. I1121 15:30:26.416659 3798194 cpu.go:282] Add success. I1121 15:30:26.442216 3798194 net.go:642] Add success. I1121 15:30:26.443798 3798194 net.go:778] primary dev: ETH0 I1121 15:30:26.443815 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:30:26.443828 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:30:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:30:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:30:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:30:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:30:36.411977 3798194 memory.go:184] no items to output this cycle I1121 15:30:36.412143 3798194 cpu.go:275] no items to output this cycle E1121 15:30:46.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:30:46.413257 3798194 memory.go:184] no items to output this cycle I1121 15:30:46.413415 3798194 cpu.go:275] no items to output this cycle E1121 15:30:56.411907 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:30:56.411940 3798194 memory.go:191] Add success. W1121 15:30:56.411969 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:30:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:30:56.411985 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:30:56.412352 3798194 cpu.go:282] Add success. I1121 15:30:56.444695 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:30:56.445182 3798194 net.go:642] Add success. I1121 15:30:56.449249 3798194 net.go:778] primary dev: ETH0 I1121 15:30:56.449264 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:30:56.449279 3798194 net.go:659] Add eni metrics success, count: 7 E1121 15:30:57.450604 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 15:30:57.450618 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:30:57.450809 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:30:57.450818 3798194 disk_worker.go:723] disk space is not compliant W1121 15:30:57.450821 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:30:57.459180 3798194 disk_worker.go:509] system disk:vda1 I1121 15:30:57.459222 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:30:57.898131 3798194 disk_info.go:125] begin check local disk info of client I1121 15:30:57.901966 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:30:57.901975 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037ef00 0xc00037ef40] E1121 15:30:58.448465 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 15:30:58.448479 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 15:30:59.448591 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 15:30:59.449668 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:30:59.449717 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:30:59.449741 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:30:59.471254 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:31:06.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:31:06.412253 3798194 memory.go:184] no items to output this cycle I1121 15:31:06.412393 3798194 cpu.go:275] no items to output this cycle E1121 15:31:16.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:31:16.412265 3798194 memory.go:184] no items to output this cycle I1121 15:31:16.412407 3798194 cpu.go:275] no items to output this cycle E1121 15:31:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:31:26.413268 3798194 memory.go:191] Add success. I1121 15:31:26.417279 3798194 cpu.go:282] Add success. I1121 15:31:26.442144 3798194 net.go:642] Add success. I1121 15:31:26.443596 3798194 net.go:778] primary dev: ETH0 I1121 15:31:26.443611 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:31:26.443625 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:31:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:31:29.450460 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:31:29.450485 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:31:36.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:31:36.411983 3798194 memory.go:184] no items to output this cycle I1121 15:31:36.412245 3798194 cpu.go:275] no items to output this cycle E1121 15:31:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:31:46.411983 3798194 memory.go:184] no items to output this cycle I1121 15:31:46.412151 3798194 cpu.go:275] no items to output this cycle E1121 15:31:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:31:56.411955 3798194 memory.go:191] Add success. W1121 15:31:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:31:56.412003 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:31:56.412006 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:31:56.412381 3798194 cpu.go:282] Add success. I1121 15:31:56.440925 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:31:56.441514 3798194 net.go:642] Add success. I1121 15:31:56.445414 3798194 net.go:778] primary dev: ETH0 I1121 15:31:56.445428 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:31:56.445442 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:31:57.448095 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:31:57.448315 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:31:57.448377 3798194 disk_worker.go:723] disk space is not compliant W1121 15:31:57.448380 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:31:57.451176 3798194 disk_worker.go:509] system disk:vda1 I1121 15:31:57.451217 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:31:57.902076 3798194 disk_info.go:125] begin check local disk info of client I1121 15:31:57.905636 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:31:57.905657 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000440d00 0xc000440d40] I1121 15:31:58.445986 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:31:59.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:31:59.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:31:59.450140 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:31:59.471649 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:32:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:32:06.413255 3798194 memory.go:184] no items to output this cycle I1121 15:32:06.413396 3798194 cpu.go:275] no items to output this cycle E1121 15:32:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:32:16.412261 3798194 memory.go:184] no items to output this cycle I1121 15:32:16.412407 3798194 cpu.go:275] no items to output this cycle E1121 15:32:26.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:32:26.411959 3798194 memory.go:191] Add success. I1121 15:32:26.412291 3798194 cpu.go:282] Add success. I1121 15:32:26.441764 3798194 net.go:642] Add success. I1121 15:32:26.445694 3798194 net.go:778] primary dev: ETH0 I1121 15:32:26.445708 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:32:26.445724 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:32:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:32:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:32:29.450125 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:32:36.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:32:36.413248 3798194 memory.go:184] no items to output this cycle I1121 15:32:36.413404 3798194 cpu.go:275] no items to output this cycle E1121 15:32:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:32:46.411976 3798194 memory.go:184] no items to output this cycle I1121 15:32:46.412138 3798194 cpu.go:275] no items to output this cycle E1121 15:32:56.412250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:32:56.412283 3798194 memory.go:191] Add success. W1121 15:32:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:32:56.412322 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:32:56.412326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:32:56.418392 3798194 cpu.go:282] Add success. I1121 15:32:56.441722 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:32:56.442221 3798194 net.go:642] Add success. I1121 15:32:56.443716 3798194 net.go:778] primary dev: ETH0 I1121 15:32:56.443731 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:32:56.443741 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:32:57.447904 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:32:57.448164 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:32:57.448176 3798194 disk_worker.go:723] disk space is not compliant W1121 15:32:57.448179 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:32:57.451027 3798194 disk_worker.go:509] system disk:vda1 I1121 15:32:57.451083 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:32:57.905762 3798194 disk_info.go:125] begin check local disk info of client I1121 15:32:57.909393 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:32:57.909402 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00049f640 0xc00049f680] I1121 15:32:58.446345 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:32:59.450386 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:32:59.450459 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:32:59.450485 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:32:59.471007 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:33:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:33:06.412255 3798194 memory.go:184] no items to output this cycle I1121 15:33:06.412399 3798194 cpu.go:275] no items to output this cycle E1121 15:33:16.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:33:16.412877 3798194 memory.go:184] no items to output this cycle I1121 15:33:16.413030 3798194 cpu.go:275] no items to output this cycle I1121 15:33:17.497584 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:33:17.497593 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:33:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:33:26.413264 3798194 memory.go:191] Add success. I1121 15:33:26.417421 3798194 cpu.go:282] Add success. I1121 15:33:26.441958 3798194 net.go:642] Add success. I1121 15:33:26.443319 3798194 net.go:778] primary dev: ETH0 I1121 15:33:26.443335 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:33:26.443350 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:33:29.450144 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:33:29.450227 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:33:29.450253 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:33:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:33:36.411984 3798194 memory.go:184] no items to output this cycle I1121 15:33:36.412134 3798194 cpu.go:275] no items to output this cycle E1121 15:33:46.413249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:33:46.413273 3798194 memory.go:184] no items to output this cycle I1121 15:33:46.413421 3798194 cpu.go:275] no items to output this cycle E1121 15:33:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:33:56.412266 3798194 memory.go:191] Add success. W1121 15:33:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:33:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:33:56.412317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:33:56.416468 3798194 cpu.go:282] Add success. I1121 15:33:56.441862 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:33:56.442357 3798194 net.go:642] Add success. I1121 15:33:56.443874 3798194 net.go:778] primary dev: ETH0 I1121 15:33:56.443888 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:33:56.443902 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:33:57.450385 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:33:57.450563 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:33:57.450619 3798194 disk_worker.go:723] disk space is not compliant W1121 15:33:57.450622 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:33:57.458897 3798194 disk_worker.go:509] system disk:vda1 I1121 15:33:57.458939 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:33:57.910135 3798194 disk_info.go:125] begin check local disk info of client I1121 15:33:57.913698 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:33:57.913707 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ec280 0xc0003ec2c0] I1121 15:33:58.446384 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:33:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:33:59.450088 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:33:59.450128 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:33:59.471676 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:34:06.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:34:06.411973 3798194 memory.go:184] no items to output this cycle I1121 15:34:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 15:34:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:34:16.411976 3798194 memory.go:184] no items to output this cycle I1121 15:34:16.412134 3798194 cpu.go:275] no items to output this cycle E1121 15:34:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:34:26.413268 3798194 memory.go:191] Add success. I1121 15:34:26.417747 3798194 cpu.go:282] Add success. I1121 15:34:26.442187 3798194 net.go:642] Add success. I1121 15:34:26.443843 3798194 net.go:778] primary dev: ETH0 I1121 15:34:26.443858 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:34:26.443871 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:34:29.450132 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:34:29.450219 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:34:29.450248 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:34:36.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:34:36.411982 3798194 memory.go:184] no items to output this cycle I1121 15:34:36.412137 3798194 cpu.go:275] no items to output this cycle E1121 15:34:46.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:34:46.412254 3798194 memory.go:184] no items to output this cycle I1121 15:34:46.412402 3798194 cpu.go:275] no items to output this cycle E1121 15:34:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:34:56.413268 3798194 memory.go:191] Add success. W1121 15:34:56.413296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:34:56.413307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:34:56.413310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:34:56.417751 3798194 cpu.go:282] Add success. I1121 15:34:56.441136 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:34:56.441616 3798194 net.go:642] Add success. I1121 15:34:56.443380 3798194 net.go:778] primary dev: ETH0 I1121 15:34:56.443394 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:34:56.443408 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:34:57.450449 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:34:57.450647 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:34:57.450658 3798194 disk_worker.go:723] disk space is not compliant W1121 15:34:57.450661 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:34:57.458685 3798194 disk_worker.go:509] system disk:vda1 I1121 15:34:57.458726 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:34:57.915145 3798194 disk_info.go:125] begin check local disk info of client I1121 15:34:57.918503 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:34:57.918512 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000273cc0 0xc000273d00] I1121 15:34:58.446198 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:34:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:34:59.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:34:59.450505 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:34:59.471040 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:35:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:35:06.412257 3798194 memory.go:184] no items to output this cycle I1121 15:35:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 15:35:16.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:35:16.412883 3798194 memory.go:184] no items to output this cycle I1121 15:35:16.413027 3798194 cpu.go:275] no items to output this cycle E1121 15:35:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:35:26.413277 3798194 memory.go:191] Add success. I1121 15:35:26.417707 3798194 cpu.go:282] Add success. I1121 15:35:26.442269 3798194 net.go:642] Add success. I1121 15:35:26.443834 3798194 net.go:778] primary dev: ETH0 I1121 15:35:26.443848 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:35:26.443861 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:35:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:35:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:35:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:35:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:35:36.411973 3798194 memory.go:184] no items to output this cycle I1121 15:35:36.412124 3798194 cpu.go:275] no items to output this cycle E1121 15:35:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:35:46.411978 3798194 memory.go:184] no items to output this cycle I1121 15:35:46.412144 3798194 cpu.go:275] no items to output this cycle E1121 15:35:56.411909 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:35:56.411942 3798194 memory.go:191] Add success. W1121 15:35:56.411968 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:35:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:35:56.411982 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:35:56.412334 3798194 cpu.go:282] Add success. I1121 15:35:56.441553 3798194 event_worker.go:152] Polling the log file for events... I1121 15:35:56.444512 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:35:56.444981 3798194 net.go:642] Add success. I1121 15:35:56.449519 3798194 net.go:778] primary dev: ETH0 I1121 15:35:56.449535 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:35:56.449548 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:35:56.449567 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795244756, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 15:35:56.459611 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 15:35:57.448928 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 15:35:57.448942 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:35:57.449180 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:35:57.449191 3798194 disk_worker.go:723] disk space is not compliant W1121 15:35:57.449194 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:35:57.450987 3798194 disk_worker.go:509] system disk:vda1 I1121 15:35:57.451026 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:35:57.918617 3798194 disk_info.go:125] begin check local disk info of client I1121 15:35:57.922220 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:35:57.922230 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030f440 0xc00030f480] E1121 15:35:58.447819 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 15:35:58.447834 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 15:35:59.448613 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 15:35:59.449660 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:35:59.449709 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:35:59.449734 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:35:59.471188 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:36:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:36:06.412258 3798194 memory.go:184] no items to output this cycle I1121 15:36:06.412401 3798194 cpu.go:275] no items to output this cycle E1121 15:36:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:36:16.411978 3798194 memory.go:184] no items to output this cycle I1121 15:36:16.412139 3798194 cpu.go:275] no items to output this cycle I1121 15:36:17.498337 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:36:17.498345 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:36:26.412841 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:36:26.412875 3798194 memory.go:191] Add success. I1121 15:36:26.417077 3798194 cpu.go:282] Add success. I1121 15:36:26.441603 3798194 net.go:642] Add success. I1121 15:36:26.443205 3798194 net.go:778] primary dev: ETH0 I1121 15:36:26.443219 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:36:26.443232 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:36:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:36:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:36:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:36:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:36:36.411975 3798194 memory.go:184] no items to output this cycle I1121 15:36:36.412141 3798194 cpu.go:275] no items to output this cycle E1121 15:36:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:36:46.413254 3798194 memory.go:184] no items to output this cycle I1121 15:36:46.413397 3798194 cpu.go:275] no items to output this cycle E1121 15:36:56.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:36:56.413272 3798194 memory.go:191] Add success. W1121 15:36:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:36:56.413320 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:36:56.413323 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:36:56.417454 3798194 cpu.go:282] Add success. I1121 15:36:56.441821 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:36:56.442323 3798194 net.go:642] Add success. I1121 15:36:56.443845 3798194 net.go:778] primary dev: ETH0 I1121 15:36:56.443859 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:36:56.443873 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:36:57.451413 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:36:57.451632 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:36:57.451642 3798194 disk_worker.go:723] disk space is not compliant W1121 15:36:57.451646 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:36:57.460172 3798194 disk_worker.go:509] system disk:vda1 I1121 15:36:57.460218 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:36:57.922322 3798194 disk_info.go:125] begin check local disk info of client I1121 15:36:57.926155 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:36:57.926164 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047d140 0xc00047d180] I1121 15:36:58.446394 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:36:59.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:36:59.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:36:59.450500 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:36:59.470732 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:37:06.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:37:06.412878 3798194 memory.go:184] no items to output this cycle I1121 15:37:06.413021 3798194 cpu.go:275] no items to output this cycle E1121 15:37:16.411971 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:37:16.411996 3798194 memory.go:184] no items to output this cycle I1121 15:37:16.412156 3798194 cpu.go:275] no items to output this cycle E1121 15:37:26.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:37:26.411948 3798194 memory.go:191] Add success. I1121 15:37:26.412298 3798194 cpu.go:282] Add success. I1121 15:37:26.442001 3798194 net.go:642] Add success. I1121 15:37:26.445826 3798194 net.go:778] primary dev: ETH0 I1121 15:37:26.445840 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:37:26.445854 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:37:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:37:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:37:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:37:36.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:37:36.412251 3798194 memory.go:184] no items to output this cycle I1121 15:37:36.412390 3798194 cpu.go:275] no items to output this cycle E1121 15:37:46.411971 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:37:46.411993 3798194 memory.go:184] no items to output this cycle I1121 15:37:46.412158 3798194 cpu.go:275] no items to output this cycle E1121 15:37:56.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:37:56.412891 3798194 memory.go:191] Add success. W1121 15:37:56.412924 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:37:56.412935 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:37:56.412939 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:37:56.417181 3798194 cpu.go:282] Add success. I1121 15:37:56.441570 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:37:56.442162 3798194 net.go:642] Add success. I1121 15:37:56.443572 3798194 net.go:778] primary dev: ETH0 I1121 15:37:56.443589 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:37:56.443601 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:37:57.450763 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:37:57.450923 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:37:57.450978 3798194 disk_worker.go:723] disk space is not compliant W1121 15:37:57.450981 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:37:57.459457 3798194 disk_worker.go:509] system disk:vda1 I1121 15:37:57.459501 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:37:57.926274 3798194 disk_info.go:125] begin check local disk info of client I1121 15:37:57.929813 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:37:57.929822 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037f400 0xc00037f440] I1121 15:37:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:37:59.450475 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:37:59.450555 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:37:59.450581 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:37:59.471100 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:38:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:38:06.412260 3798194 memory.go:184] no items to output this cycle I1121 15:38:06.412406 3798194 cpu.go:275] no items to output this cycle E1121 15:38:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:38:16.411977 3798194 memory.go:184] no items to output this cycle I1121 15:38:16.412130 3798194 cpu.go:275] no items to output this cycle E1121 15:38:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:38:26.411953 3798194 memory.go:191] Add success. I1121 15:38:26.412292 3798194 cpu.go:282] Add success. I1121 15:38:26.441961 3798194 net.go:642] Add success. I1121 15:38:26.446294 3798194 net.go:778] primary dev: ETH0 I1121 15:38:26.446310 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:38:26.446323 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:38:29.450407 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:38:29.450487 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:38:29.450517 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:38:36.412966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:38:36.412990 3798194 memory.go:184] no items to output this cycle I1121 15:38:36.413153 3798194 cpu.go:275] no items to output this cycle E1121 15:38:46.411863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:38:46.411887 3798194 memory.go:184] no items to output this cycle I1121 15:38:46.412035 3798194 cpu.go:275] no items to output this cycle E1121 15:38:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:38:56.411958 3798194 memory.go:191] Add success. W1121 15:38:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:38:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:38:56.412008 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:38:56.412354 3798194 cpu.go:282] Add success. I1121 15:38:56.441831 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:38:56.442333 3798194 net.go:642] Add success. I1121 15:38:56.446082 3798194 net.go:778] primary dev: ETH0 I1121 15:38:56.446097 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:38:56.446121 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:38:57.450560 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:38:57.450745 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:38:57.450800 3798194 disk_worker.go:723] disk space is not compliant W1121 15:38:57.450803 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:38:57.459342 3798194 disk_worker.go:509] system disk:vda1 I1121 15:38:57.459383 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:38:57.929923 3798194 disk_info.go:125] begin check local disk info of client I1121 15:38:57.933782 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:38:57.933791 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037ec00 0xc00037ec40] I1121 15:38:58.446418 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:38:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:38:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:38:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:38:59.471667 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:39:06.412764 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:39:06.412788 3798194 memory.go:184] no items to output this cycle I1121 15:39:06.412941 3798194 cpu.go:275] no items to output this cycle E1121 15:39:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:39:16.411971 3798194 memory.go:184] no items to output this cycle I1121 15:39:16.412131 3798194 cpu.go:275] no items to output this cycle I1121 15:39:17.499265 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:39:17.499273 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:39:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:39:26.412267 3798194 memory.go:191] Add success. I1121 15:39:26.416959 3798194 cpu.go:282] Add success. I1121 15:39:26.441518 3798194 net.go:642] Add success. I1121 15:39:26.442989 3798194 net.go:778] primary dev: ETH0 I1121 15:39:26.443004 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:39:26.443018 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:39:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:39:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:39:29.450129 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:39:36.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:39:36.412253 3798194 memory.go:184] no items to output this cycle I1121 15:39:36.412404 3798194 cpu.go:275] no items to output this cycle E1121 15:39:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:39:46.411984 3798194 memory.go:184] no items to output this cycle I1121 15:39:46.412139 3798194 cpu.go:275] no items to output this cycle E1121 15:39:56.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:39:56.412278 3798194 memory.go:191] Add success. W1121 15:39:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:39:56.412317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:39:56.412321 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:39:56.416991 3798194 cpu.go:282] Add success. I1121 15:39:56.441377 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:39:56.441937 3798194 net.go:642] Add success. I1121 15:39:56.443477 3798194 net.go:778] primary dev: ETH0 I1121 15:39:56.443492 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:39:56.443657 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:39:57.450638 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:39:57.450798 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:39:57.450852 3798194 disk_worker.go:723] disk space is not compliant W1121 15:39:57.450855 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:39:57.458935 3798194 disk_worker.go:509] system disk:vda1 I1121 15:39:57.458975 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:39:57.933898 3798194 disk_info.go:125] begin check local disk info of client I1121 15:39:57.937446 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:39:57.937457 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004c2180 0xc0004c21c0] I1121 15:39:58.445994 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:39:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:39:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:39:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:39:59.471657 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:40:06.412880 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:40:06.412903 3798194 memory.go:184] no items to output this cycle I1121 15:40:06.413044 3798194 cpu.go:275] no items to output this cycle E1121 15:40:16.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:40:16.412878 3798194 memory.go:184] no items to output this cycle I1121 15:40:16.413033 3798194 cpu.go:275] no items to output this cycle E1121 15:40:26.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:40:26.412276 3798194 memory.go:191] Add success. I1121 15:40:26.416521 3798194 cpu.go:282] Add success. I1121 15:40:26.442072 3798194 net.go:642] Add success. I1121 15:40:26.443572 3798194 net.go:778] primary dev: ETH0 I1121 15:40:26.443587 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:40:26.443601 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:40:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:40:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:40:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:40:36.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:40:36.413250 3798194 memory.go:184] no items to output this cycle I1121 15:40:36.413391 3798194 cpu.go:275] no items to output this cycle E1121 15:40:46.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:40:46.412263 3798194 memory.go:184] no items to output this cycle I1121 15:40:46.412409 3798194 cpu.go:275] no items to output this cycle E1121 15:40:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:40:56.411956 3798194 memory.go:191] Add success. W1121 15:40:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:40:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:40:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:40:56.412332 3798194 cpu.go:282] Add success. I1121 15:40:56.444465 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:40:56.444945 3798194 net.go:642] Add success. I1121 15:40:56.448974 3798194 net.go:778] primary dev: ETH0 I1121 15:40:56.448988 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:40:56.449002 3798194 net.go:659] Add eni metrics success, count: 7 E1121 15:40:57.450294 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 15:40:57.450317 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:40:57.450466 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:40:57.450516 3798194 disk_worker.go:723] disk space is not compliant W1121 15:40:57.450519 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:40:57.458569 3798194 disk_worker.go:509] system disk:vda1 I1121 15:40:57.458613 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:40:57.937557 3798194 disk_info.go:125] begin check local disk info of client I1121 15:40:57.941048 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:40:57.941056 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ff400 0xc0003ff440] E1121 15:40:58.447812 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 15:40:58.447826 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 15:40:59.448585 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 15:40:59.449662 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:40:59.449722 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:40:59.449743 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:40:59.471258 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:41:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:41:06.412264 3798194 memory.go:184] no items to output this cycle I1121 15:41:06.412408 3798194 cpu.go:275] no items to output this cycle E1121 15:41:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:41:16.412258 3798194 memory.go:184] no items to output this cycle I1121 15:41:16.412405 3798194 cpu.go:275] no items to output this cycle E1121 15:41:26.411937 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:41:26.411975 3798194 memory.go:191] Add success. I1121 15:41:26.412356 3798194 cpu.go:282] Add success. I1121 15:41:26.441978 3798194 net.go:642] Add success. I1121 15:41:26.446524 3798194 net.go:778] primary dev: ETH0 I1121 15:41:26.446540 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:41:26.446551 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:41:29.450034 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:41:29.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:41:29.450150 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:41:36.412777 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:41:36.412801 3798194 memory.go:184] no items to output this cycle I1121 15:41:36.412953 3798194 cpu.go:275] no items to output this cycle E1121 15:41:46.411973 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:41:46.411994 3798194 memory.go:184] no items to output this cycle I1121 15:41:46.412156 3798194 cpu.go:275] no items to output this cycle E1121 15:41:56.412223 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:41:56.412257 3798194 memory.go:191] Add success. W1121 15:41:56.412291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:41:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:41:56.412307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:41:56.417333 3798194 cpu.go:282] Add success. I1121 15:41:56.441746 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:41:56.442355 3798194 net.go:642] Add success. I1121 15:41:56.443963 3798194 net.go:778] primary dev: ETH0 I1121 15:41:56.443978 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:41:56.443990 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:41:57.450924 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:41:57.451145 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:41:57.451156 3798194 disk_worker.go:723] disk space is not compliant W1121 15:41:57.451159 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:41:57.459482 3798194 disk_worker.go:509] system disk:vda1 I1121 15:41:57.459524 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:41:57.941169 3798194 disk_info.go:125] begin check local disk info of client I1121 15:41:57.944678 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:41:57.944686 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030b300 0xc00030b340] I1121 15:41:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:41:59.450036 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:41:59.450126 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:41:59.450152 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:41:59.471685 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:42:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:42:06.411984 3798194 memory.go:184] no items to output this cycle I1121 15:42:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 15:42:16.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:42:16.412260 3798194 memory.go:184] no items to output this cycle I1121 15:42:16.412405 3798194 cpu.go:275] no items to output this cycle I1121 15:42:17.500356 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:42:17.500365 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:42:26.411275 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:42:26.411310 3798194 memory.go:191] Add success. I1121 15:42:26.411645 3798194 cpu.go:282] Add success. I1121 15:42:26.442246 3798194 net.go:642] Add success. I1121 15:42:26.446576 3798194 net.go:778] primary dev: ETH0 I1121 15:42:26.446591 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:42:26.446607 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:42:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:42:29.450463 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:42:29.450486 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:42:36.412811 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:42:36.412834 3798194 memory.go:184] no items to output this cycle I1121 15:42:36.412988 3798194 cpu.go:275] no items to output this cycle E1121 15:42:46.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:42:46.413257 3798194 memory.go:184] no items to output this cycle I1121 15:42:46.413408 3798194 cpu.go:275] no items to output this cycle E1121 15:42:56.412118 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:42:56.412156 3798194 memory.go:191] Add success. W1121 15:42:56.412192 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:42:56.412281 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:42:56.412285 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:42:56.412978 3798194 cpu.go:282] Add success. I1121 15:42:56.440962 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:42:56.441546 3798194 net.go:642] Add success. I1121 15:42:56.445429 3798194 net.go:778] primary dev: ETH0 I1121 15:42:56.445444 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:42:56.445457 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:42:57.450300 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:42:57.450508 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:42:57.450519 3798194 disk_worker.go:723] disk space is not compliant W1121 15:42:57.450522 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:42:57.458561 3798194 disk_worker.go:509] system disk:vda1 I1121 15:42:57.458604 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:42:57.946146 3798194 disk_info.go:125] begin check local disk info of client I1121 15:42:57.949686 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:42:57.949695 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030eb80 0xc00030ebc0] I1121 15:42:58.446390 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:42:59.450487 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:42:59.450557 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:42:59.450583 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:42:59.471099 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:43:06.411973 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:43:06.411997 3798194 memory.go:184] no items to output this cycle I1121 15:43:06.412169 3798194 cpu.go:275] no items to output this cycle E1121 15:43:16.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:43:16.412267 3798194 memory.go:184] no items to output this cycle I1121 15:43:16.412420 3798194 cpu.go:275] no items to output this cycle E1121 15:43:26.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:43:26.411947 3798194 memory.go:191] Add success. I1121 15:43:26.412302 3798194 cpu.go:282] Add success. I1121 15:43:26.441911 3798194 net.go:642] Add success. I1121 15:43:26.446304 3798194 net.go:778] primary dev: ETH0 I1121 15:43:26.446320 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:43:26.446335 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:43:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:43:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:43:29.450496 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:43:36.413163 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:43:36.413185 3798194 memory.go:184] no items to output this cycle I1121 15:43:36.413329 3798194 cpu.go:275] no items to output this cycle E1121 15:43:46.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:43:46.412874 3798194 memory.go:184] no items to output this cycle I1121 15:43:46.413021 3798194 cpu.go:275] no items to output this cycle E1121 15:43:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:43:56.411958 3798194 memory.go:191] Add success. W1121 15:43:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:43:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:43:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:43:56.412336 3798194 cpu.go:282] Add success. I1121 15:43:56.441853 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:43:56.442426 3798194 net.go:642] Add success. I1121 15:43:56.446573 3798194 net.go:778] primary dev: ETH0 I1121 15:43:56.446592 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:43:56.446605 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:43:57.450286 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:43:57.450488 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:43:57.450498 3798194 disk_worker.go:723] disk space is not compliant W1121 15:43:57.450501 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:43:57.458612 3798194 disk_worker.go:509] system disk:vda1 I1121 15:43:57.458668 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:43:57.949799 3798194 disk_info.go:125] begin check local disk info of client I1121 15:43:57.953452 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:43:57.953461 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004024c0 0xc000402500] I1121 15:43:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:43:59.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:43:59.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:43:59.450512 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:43:59.471070 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:44:06.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:44:06.411985 3798194 memory.go:184] no items to output this cycle I1121 15:44:06.412143 3798194 cpu.go:275] no items to output this cycle E1121 15:44:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:44:16.411982 3798194 memory.go:184] no items to output this cycle I1121 15:44:16.412149 3798194 cpu.go:275] no items to output this cycle E1121 15:44:26.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:44:26.411949 3798194 memory.go:191] Add success. I1121 15:44:26.412298 3798194 cpu.go:282] Add success. I1121 15:44:26.441732 3798194 net.go:642] Add success. I1121 15:44:26.445720 3798194 net.go:778] primary dev: ETH0 I1121 15:44:26.445742 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:44:26.445762 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:44:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:44:29.450086 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:44:29.450121 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:44:36.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:44:36.413260 3798194 memory.go:184] no items to output this cycle I1121 15:44:36.413401 3798194 cpu.go:275] no items to output this cycle E1121 15:44:46.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:44:46.412253 3798194 memory.go:184] no items to output this cycle I1121 15:44:46.412400 3798194 cpu.go:275] no items to output this cycle E1121 15:44:56.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:44:56.412286 3798194 memory.go:191] Add success. W1121 15:44:56.412317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:44:56.412329 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:44:56.412332 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:44:56.416511 3798194 cpu.go:282] Add success. I1121 15:44:56.441885 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:44:56.442391 3798194 net.go:642] Add success. I1121 15:44:56.443887 3798194 net.go:778] primary dev: ETH0 I1121 15:44:56.443902 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:44:56.443913 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:44:57.450296 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:44:57.450457 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:44:57.450514 3798194 disk_worker.go:723] disk space is not compliant W1121 15:44:57.450517 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:44:57.458622 3798194 disk_worker.go:509] system disk:vda1 I1121 15:44:57.458665 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:44:57.955134 3798194 disk_info.go:125] begin check local disk info of client I1121 15:44:57.958592 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:44:57.958601 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d2140 0xc0003d2180] I1121 15:44:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:44:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:44:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:44:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:44:59.471658 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:45:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:45:06.411979 3798194 memory.go:184] no items to output this cycle I1121 15:45:06.412131 3798194 cpu.go:275] no items to output this cycle E1121 15:45:16.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:45:16.411972 3798194 memory.go:184] no items to output this cycle I1121 15:45:16.412140 3798194 cpu.go:275] no items to output this cycle I1121 15:45:17.500982 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:45:17.500991 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:45:26.411215 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:45:26.411251 3798194 memory.go:191] Add success. I1121 15:45:26.411721 3798194 cpu.go:282] Add success. I1121 15:45:26.441299 3798194 net.go:642] Add success. I1121 15:45:26.445102 3798194 net.go:778] primary dev: ETH0 I1121 15:45:26.445130 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:45:26.445144 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:45:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:45:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:45:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:45:36.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:45:36.412262 3798194 memory.go:184] no items to output this cycle I1121 15:45:36.412413 3798194 cpu.go:275] no items to output this cycle E1121 15:45:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:45:46.411983 3798194 memory.go:184] no items to output this cycle I1121 15:45:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 15:45:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:45:56.412268 3798194 memory.go:191] Add success. W1121 15:45:56.412301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:45:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:45:56.412316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:45:56.416674 3798194 cpu.go:282] Add success. I1121 15:45:56.443612 3798194 event_worker.go:152] Polling the log file for events... I1121 15:45:56.443838 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:45:56.444395 3798194 net.go:642] Add success. I1121 15:45:56.445890 3798194 net.go:778] primary dev: ETH0 I1121 15:45:56.445904 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:45:56.445916 3798194 net.go:659] Add eni metrics success, count: 7 E1121 15:45:57.451349 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 15:45:57.451363 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:45:57.451566 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:45:57.451575 3798194 disk_worker.go:723] disk space is not compliant W1121 15:45:57.451578 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:45:57.460015 3798194 disk_worker.go:509] system disk:vda1 I1121 15:45:57.460061 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:45:57.958701 3798194 disk_info.go:125] begin check local disk info of client I1121 15:45:57.962507 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:45:57.962516 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00043c640 0xc00043c680] E1121 15:45:58.448487 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 15:45:58.448502 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 15:45:59.449127 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 15:45:59.450204 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:45:59.450254 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:45:59.450273 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:45:59.470730 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:46:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:46:06.411976 3798194 memory.go:184] no items to output this cycle I1121 15:46:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 15:46:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:46:16.411975 3798194 memory.go:184] no items to output this cycle I1121 15:46:16.412127 3798194 cpu.go:275] no items to output this cycle E1121 15:46:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:46:26.413270 3798194 memory.go:191] Add success. I1121 15:46:26.417897 3798194 cpu.go:282] Add success. I1121 15:46:26.441448 3798194 net.go:642] Add success. I1121 15:46:26.443184 3798194 net.go:778] primary dev: ETH0 I1121 15:46:26.443199 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:46:26.443211 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:46:29.450034 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:46:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:46:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:46:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:46:36.411978 3798194 memory.go:184] no items to output this cycle I1121 15:46:36.412131 3798194 cpu.go:275] no items to output this cycle E1121 15:46:46.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:46:46.411989 3798194 memory.go:184] no items to output this cycle I1121 15:46:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 15:46:56.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:46:56.412284 3798194 memory.go:191] Add success. W1121 15:46:56.412314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:46:56.412325 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:46:56.412328 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:46:56.416531 3798194 cpu.go:282] Add success. I1121 15:46:56.441928 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:46:56.442495 3798194 net.go:642] Add success. I1121 15:46:56.443832 3798194 net.go:778] primary dev: ETH0 I1121 15:46:56.443847 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:46:56.443861 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:46:57.451194 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:46:57.451391 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:46:57.451404 3798194 disk_worker.go:723] disk space is not compliant W1121 15:46:57.451407 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:46:57.459848 3798194 disk_worker.go:509] system disk:vda1 I1121 15:46:57.459890 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:46:57.964146 3798194 disk_info.go:125] begin check local disk info of client I1121 15:46:57.967927 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:46:57.967936 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0280 0xc0002e02c0] I1121 15:46:58.446391 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:46:59.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:46:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:46:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:46:59.471662 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:47:06.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:47:06.411981 3798194 memory.go:184] no items to output this cycle I1121 15:47:06.412139 3798194 cpu.go:275] no items to output this cycle E1121 15:47:16.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:47:16.412885 3798194 memory.go:184] no items to output this cycle I1121 15:47:16.413047 3798194 cpu.go:275] no items to output this cycle E1121 15:47:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:47:26.411959 3798194 memory.go:191] Add success. I1121 15:47:26.412303 3798194 cpu.go:282] Add success. I1121 15:47:26.442012 3798194 net.go:642] Add success. I1121 15:47:26.446190 3798194 net.go:778] primary dev: ETH0 I1121 15:47:26.446206 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:47:26.446218 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:47:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:47:29.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:47:29.450148 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:47:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:47:36.412253 3798194 memory.go:184] no items to output this cycle I1121 15:47:36.412398 3798194 cpu.go:275] no items to output this cycle E1121 15:47:46.411944 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:47:46.411966 3798194 memory.go:184] no items to output this cycle I1121 15:47:46.412125 3798194 cpu.go:275] no items to output this cycle E1121 15:47:56.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:47:56.412892 3798194 memory.go:191] Add success. W1121 15:47:56.412918 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:47:56.412931 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:47:56.412934 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:47:56.417238 3798194 cpu.go:282] Add success. I1121 15:47:56.441626 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:47:56.442129 3798194 net.go:642] Add success. I1121 15:47:56.443715 3798194 net.go:778] primary dev: ETH0 I1121 15:47:56.443729 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:47:56.443743 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:47:57.450595 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:47:57.450792 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:47:57.450802 3798194 disk_worker.go:723] disk space is not compliant W1121 15:47:57.450805 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:47:57.459997 3798194 disk_worker.go:509] system disk:vda1 I1121 15:47:57.460034 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:47:57.968064 3798194 disk_info.go:125] begin check local disk info of client I1121 15:47:57.971950 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:47:57.971960 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000447740 0xc000447780] I1121 15:47:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:47:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:47:59.450088 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:47:59.450124 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:47:59.471662 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:48:06.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:48:06.413254 3798194 memory.go:184] no items to output this cycle I1121 15:48:06.413397 3798194 cpu.go:275] no items to output this cycle E1121 15:48:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:48:16.411976 3798194 memory.go:184] no items to output this cycle I1121 15:48:16.412143 3798194 cpu.go:275] no items to output this cycle I1121 15:48:17.501211 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:48:17.501220 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:48:26.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:48:26.413267 3798194 memory.go:191] Add success. I1121 15:48:26.417471 3798194 cpu.go:282] Add success. I1121 15:48:26.441986 3798194 net.go:642] Add success. I1121 15:48:26.443657 3798194 net.go:778] primary dev: ETH0 I1121 15:48:26.443671 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:48:26.443685 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:48:29.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:48:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:48:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:48:36.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:48:36.411977 3798194 memory.go:184] no items to output this cycle I1121 15:48:36.412134 3798194 cpu.go:275] no items to output this cycle E1121 15:48:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:48:46.411974 3798194 memory.go:184] no items to output this cycle I1121 15:48:46.412131 3798194 cpu.go:275] no items to output this cycle E1121 15:48:56.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:48:56.412885 3798194 memory.go:191] Add success. W1121 15:48:56.412927 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:48:56.412939 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:48:56.412942 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:48:56.417131 3798194 cpu.go:282] Add success. I1121 15:48:56.441423 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:48:56.441909 3798194 net.go:642] Add success. I1121 15:48:56.443293 3798194 net.go:778] primary dev: ETH0 I1121 15:48:56.443307 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:48:56.443319 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:48:57.450353 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:48:57.450518 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:48:57.450574 3798194 disk_worker.go:723] disk space is not compliant W1121 15:48:57.450578 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:48:57.458773 3798194 disk_worker.go:509] system disk:vda1 I1121 15:48:57.458816 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:48:57.974134 3798194 disk_info.go:125] begin check local disk info of client I1121 15:48:57.977739 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:48:57.977748 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c4200 0xc0003c4240] I1121 15:48:58.445996 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:48:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:48:59.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:48:59.450494 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:48:59.471018 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:49:06.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:49:06.412247 3798194 memory.go:184] no items to output this cycle I1121 15:49:06.412396 3798194 cpu.go:275] no items to output this cycle E1121 15:49:16.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:49:16.412268 3798194 memory.go:184] no items to output this cycle I1121 15:49:16.412418 3798194 cpu.go:275] no items to output this cycle E1121 15:49:26.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:49:26.412048 3798194 memory.go:191] Add success. I1121 15:49:26.412648 3798194 cpu.go:282] Add success. I1121 15:49:26.441320 3798194 net.go:642] Add success. I1121 15:49:26.445407 3798194 net.go:778] primary dev: ETH0 I1121 15:49:26.445422 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:49:26.445434 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:49:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:49:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:49:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:49:36.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:49:36.411981 3798194 memory.go:184] no items to output this cycle I1121 15:49:36.412133 3798194 cpu.go:275] no items to output this cycle E1121 15:49:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:49:46.411980 3798194 memory.go:184] no items to output this cycle I1121 15:49:46.412133 3798194 cpu.go:275] no items to output this cycle E1121 15:49:56.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:49:56.412886 3798194 memory.go:191] Add success. W1121 15:49:56.412915 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:49:56.412928 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:49:56.412931 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:49:56.417283 3798194 cpu.go:282] Add success. I1121 15:49:56.441652 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:49:56.442144 3798194 net.go:642] Add success. I1121 15:49:56.443670 3798194 net.go:778] primary dev: ETH0 I1121 15:49:56.443685 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:49:56.443697 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:49:57.451049 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:49:57.451283 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:49:57.451297 3798194 disk_worker.go:723] disk space is not compliant W1121 15:49:57.451300 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:49:57.460187 3798194 disk_worker.go:509] system disk:vda1 I1121 15:49:57.460230 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:49:57.977851 3798194 disk_info.go:125] begin check local disk info of client I1121 15:49:57.981711 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:49:57.981720 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000306780 0xc0003067c0] I1121 15:49:58.446398 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:49:59.450496 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:49:59.450582 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:49:59.450606 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:49:59.471127 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:50:06.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:50:06.412882 3798194 memory.go:184] no items to output this cycle I1121 15:50:06.413026 3798194 cpu.go:275] no items to output this cycle E1121 15:50:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:50:16.411983 3798194 memory.go:184] no items to output this cycle I1121 15:50:16.412145 3798194 cpu.go:275] no items to output this cycle E1121 15:50:26.411236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:50:26.411269 3798194 memory.go:191] Add success. I1121 15:50:26.415384 3798194 cpu.go:282] Add success. I1121 15:50:26.441943 3798194 net.go:642] Add success. I1121 15:50:26.443385 3798194 net.go:778] primary dev: ETH0 I1121 15:50:26.443399 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:50:26.443412 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:50:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:50:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:50:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:50:36.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:50:36.412250 3798194 memory.go:184] no items to output this cycle I1121 15:50:36.412394 3798194 cpu.go:275] no items to output this cycle E1121 15:50:46.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:50:46.412247 3798194 memory.go:184] no items to output this cycle I1121 15:50:46.412399 3798194 cpu.go:275] no items to output this cycle E1121 15:50:56.411936 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:50:56.411968 3798194 memory.go:191] Add success. W1121 15:50:56.412008 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:50:56.412020 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:50:56.412023 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:50:56.412671 3798194 cpu.go:282] Add success. I1121 15:50:56.443578 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:50:56.444160 3798194 net.go:642] Add success. I1121 15:50:56.448365 3798194 net.go:778] primary dev: ETH0 I1121 15:50:56.448380 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:50:56.448393 3798194 net.go:659] Add eni metrics success, count: 7 E1121 15:50:57.452628 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 15:50:57.452646 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:50:57.452954 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:50:57.452969 3798194 disk_worker.go:723] disk space is not compliant W1121 15:50:57.452974 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:50:57.461248 3798194 disk_worker.go:509] system disk:vda1 I1121 15:50:57.461289 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:50:57.981825 3798194 disk_info.go:125] begin check local disk info of client I1121 15:50:57.985429 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:50:57.985438 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e040 0xc00030e080] E1121 15:50:58.447789 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 15:50:58.447803 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 15:50:59.448609 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 15:50:59.449685 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:50:59.449734 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:50:59.449752 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:50:59.471258 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:51:06.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:51:06.412264 3798194 memory.go:184] no items to output this cycle I1121 15:51:06.412410 3798194 cpu.go:275] no items to output this cycle E1121 15:51:16.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:51:16.411972 3798194 memory.go:184] no items to output this cycle I1121 15:51:16.412140 3798194 cpu.go:275] no items to output this cycle I1121 15:51:17.502354 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:51:17.502363 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:51:26.411293 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:51:26.411329 3798194 memory.go:191] Add success. I1121 15:51:26.411662 3798194 cpu.go:282] Add success. I1121 15:51:26.441293 3798194 net.go:642] Add success. I1121 15:51:26.445612 3798194 net.go:778] primary dev: ETH0 I1121 15:51:26.445628 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:51:26.445641 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:51:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:51:29.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:51:29.450123 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:51:36.412345 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:51:36.412368 3798194 memory.go:184] no items to output this cycle I1121 15:51:36.412518 3798194 cpu.go:275] no items to output this cycle E1121 15:51:46.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:51:46.411985 3798194 memory.go:184] no items to output this cycle I1121 15:51:46.412142 3798194 cpu.go:275] no items to output this cycle E1121 15:51:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:51:56.411949 3798194 memory.go:191] Add success. W1121 15:51:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:51:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:51:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:51:56.412346 3798194 cpu.go:282] Add success. I1121 15:51:56.441802 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:51:56.442319 3798194 net.go:642] Add success. I1121 15:51:56.446429 3798194 net.go:778] primary dev: ETH0 I1121 15:51:56.446443 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:51:56.446459 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:51:57.450838 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:51:57.451040 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:51:57.451051 3798194 disk_worker.go:723] disk space is not compliant W1121 15:51:57.451054 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:51:57.459890 3798194 disk_worker.go:509] system disk:vda1 I1121 15:51:57.459934 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:51:57.985547 3798194 disk_info.go:125] begin check local disk info of client I1121 15:51:57.989450 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:51:57.989460 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000410140 0xc000410180] I1121 15:51:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:51:59.450105 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:51:59.450197 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:51:59.450224 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:51:59.470701 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:52:06.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:52:06.412265 3798194 memory.go:184] no items to output this cycle I1121 15:52:06.412406 3798194 cpu.go:275] no items to output this cycle E1121 15:52:16.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:52:16.412254 3798194 memory.go:184] no items to output this cycle I1121 15:52:16.412400 3798194 cpu.go:275] no items to output this cycle E1121 15:52:26.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:52:26.411949 3798194 memory.go:191] Add success. I1121 15:52:26.412277 3798194 cpu.go:282] Add success. I1121 15:52:26.441757 3798194 net.go:642] Add success. I1121 15:52:26.445974 3798194 net.go:778] primary dev: ETH0 I1121 15:52:26.445988 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:52:26.446000 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:52:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:52:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:52:29.450147 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:52:36.412250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:52:36.412276 3798194 memory.go:184] no items to output this cycle I1121 15:52:36.412421 3798194 cpu.go:275] no items to output this cycle E1121 15:52:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:52:46.411988 3798194 memory.go:184] no items to output this cycle I1121 15:52:46.412148 3798194 cpu.go:275] no items to output this cycle E1121 15:52:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:52:56.411949 3798194 memory.go:191] Add success. W1121 15:52:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:52:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:52:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:52:56.412386 3798194 cpu.go:282] Add success. I1121 15:52:56.441883 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:52:56.442472 3798194 net.go:642] Add success. I1121 15:52:56.446772 3798194 net.go:778] primary dev: ETH0 I1121 15:52:56.446789 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:52:56.446802 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:52:57.451490 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:52:57.451711 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:52:57.451723 3798194 disk_worker.go:723] disk space is not compliant W1121 15:52:57.451726 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:52:57.460069 3798194 disk_worker.go:509] system disk:vda1 I1121 15:52:57.460124 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:52:57.991139 3798194 disk_info.go:125] begin check local disk info of client I1121 15:52:57.994957 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:52:57.994966 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000399740 0xc000399780] I1121 15:52:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:52:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:52:59.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:52:59.450147 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:52:59.471676 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:53:06.413059 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:53:06.413082 3798194 memory.go:184] no items to output this cycle I1121 15:53:06.413252 3798194 cpu.go:275] no items to output this cycle E1121 15:53:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:53:16.412257 3798194 memory.go:184] no items to output this cycle I1121 15:53:16.412401 3798194 cpu.go:275] no items to output this cycle E1121 15:53:26.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:53:26.412884 3798194 memory.go:191] Add success. I1121 15:53:26.416917 3798194 cpu.go:282] Add success. I1121 15:53:26.441599 3798194 net.go:642] Add success. I1121 15:53:26.444351 3798194 net.go:778] primary dev: ETH0 I1121 15:53:26.444367 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:53:26.444383 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:53:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:53:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:53:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:53:36.411236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:53:36.411268 3798194 memory.go:184] no items to output this cycle I1121 15:53:36.411414 3798194 cpu.go:275] no items to output this cycle E1121 15:53:46.411981 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:53:46.412009 3798194 memory.go:184] no items to output this cycle I1121 15:53:46.412178 3798194 cpu.go:275] no items to output this cycle E1121 15:53:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:53:56.411951 3798194 memory.go:191] Add success. W1121 15:53:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:53:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:53:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:53:56.412370 3798194 cpu.go:282] Add success. I1121 15:53:56.441909 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:53:56.442519 3798194 net.go:642] Add success. I1121 15:53:56.446927 3798194 net.go:778] primary dev: ETH0 I1121 15:53:56.446943 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:53:56.446955 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:53:57.450383 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:53:57.450598 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:53:57.450609 3798194 disk_worker.go:723] disk space is not compliant W1121 15:53:57.450613 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:53:57.458696 3798194 disk_worker.go:509] system disk:vda1 I1121 15:53:57.458738 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:53:57.995069 3798194 disk_info.go:125] begin check local disk info of client I1121 15:53:57.998636 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:53:57.998646 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000399940 0xc000399980] I1121 15:53:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:53:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:53:59.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:53:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:53:59.471650 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:54:06.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:54:06.413257 3798194 memory.go:184] no items to output this cycle I1121 15:54:06.413398 3798194 cpu.go:275] no items to output this cycle E1121 15:54:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:54:16.411976 3798194 memory.go:184] no items to output this cycle I1121 15:54:16.412134 3798194 cpu.go:275] no items to output this cycle I1121 15:54:17.503362 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:54:17.503370 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:54:26.412285 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:54:26.412321 3798194 memory.go:191] Add success. I1121 15:54:26.416509 3798194 cpu.go:282] Add success. I1121 15:54:26.442037 3798194 net.go:642] Add success. I1121 15:54:26.443650 3798194 net.go:778] primary dev: ETH0 I1121 15:54:26.443664 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:54:26.443679 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:54:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:54:29.450463 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:54:29.450489 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:54:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:54:36.411976 3798194 memory.go:184] no items to output this cycle I1121 15:54:36.412128 3798194 cpu.go:275] no items to output this cycle E1121 15:54:46.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:54:46.412259 3798194 memory.go:184] no items to output this cycle I1121 15:54:46.412409 3798194 cpu.go:275] no items to output this cycle E1121 15:54:56.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:54:56.412892 3798194 memory.go:191] Add success. W1121 15:54:56.412920 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:54:56.412931 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:54:56.412935 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:54:56.417315 3798194 cpu.go:282] Add success. I1121 15:54:56.441941 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:54:56.442904 3798194 net.go:642] Add success. I1121 15:54:56.444561 3798194 net.go:778] primary dev: ETH0 I1121 15:54:56.444577 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:54:56.444594 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:54:57.450797 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:54:57.450994 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:54:57.451005 3798194 disk_worker.go:723] disk space is not compliant W1121 15:54:57.451008 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:54:57.459211 3798194 disk_worker.go:509] system disk:vda1 I1121 15:54:57.459250 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:54:57.998750 3798194 disk_info.go:125] begin check local disk info of client I1121 15:54:58.002653 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:54:58.002662 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004db940 0xc0004db980] I1121 15:54:58.446397 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:54:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:54:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:54:59.450132 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:54:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:55:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:55:06.411982 3798194 memory.go:184] no items to output this cycle I1121 15:55:06.412139 3798194 cpu.go:275] no items to output this cycle E1121 15:55:16.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:55:16.412253 3798194 memory.go:184] no items to output this cycle I1121 15:55:16.412409 3798194 cpu.go:275] no items to output this cycle E1121 15:55:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:55:26.411952 3798194 memory.go:191] Add success. I1121 15:55:26.412295 3798194 cpu.go:282] Add success. I1121 15:55:26.442001 3798194 net.go:642] Add success. I1121 15:55:26.446251 3798194 net.go:778] primary dev: ETH0 I1121 15:55:26.446268 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:55:26.446282 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:55:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:55:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:55:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:55:36.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:55:36.412266 3798194 memory.go:184] no items to output this cycle I1121 15:55:36.412411 3798194 cpu.go:275] no items to output this cycle E1121 15:55:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:55:46.412254 3798194 memory.go:184] no items to output this cycle I1121 15:55:46.412394 3798194 cpu.go:275] no items to output this cycle E1121 15:55:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:55:56.411960 3798194 memory.go:191] Add success. W1121 15:55:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:55:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:55:56.412007 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:55:56.412359 3798194 cpu.go:282] Add success. I1121 15:55:56.441555 3798194 event_worker.go:152] Polling the log file for events... I1121 15:55:56.444367 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:55:56.444851 3798194 net.go:642] Add success. I1121 15:55:56.449009 3798194 net.go:778] primary dev: ETH0 I1121 15:55:56.449023 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:55:56.449036 3798194 net.go:659] Add eni metrics success, count: 7 E1121 15:55:57.450305 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 15:55:57.450319 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:55:57.450521 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:55:57.450532 3798194 disk_worker.go:723] disk space is not compliant W1121 15:55:57.450535 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:55:57.458552 3798194 disk_worker.go:509] system disk:vda1 I1121 15:55:57.458593 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:55:58.002758 3798194 disk_info.go:125] begin check local disk info of client I1121 15:55:58.006324 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:55:58.006334 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ee900 0xc0003ee940] E1121 15:55:58.447763 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 15:55:58.447779 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 15:55:59.448597 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 15:55:59.449673 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:55:59.449724 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:55:59.449755 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:55:59.471186 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:56:06.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:56:06.412267 3798194 memory.go:184] no items to output this cycle I1121 15:56:06.412623 3798194 cpu.go:275] no items to output this cycle E1121 15:56:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:56:16.411985 3798194 memory.go:184] no items to output this cycle I1121 15:56:16.412148 3798194 cpu.go:275] no items to output this cycle E1121 15:56:26.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:56:26.412890 3798194 memory.go:191] Add success. I1121 15:56:26.417197 3798194 cpu.go:282] Add success. I1121 15:56:26.441672 3798194 net.go:642] Add success. I1121 15:56:26.443185 3798194 net.go:778] primary dev: ETH0 I1121 15:56:26.443201 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:56:26.443214 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:56:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:56:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:56:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:56:36.416241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:56:36.416263 3798194 memory.go:184] no items to output this cycle I1121 15:56:36.416403 3798194 cpu.go:275] no items to output this cycle E1121 15:56:46.411976 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:56:46.411999 3798194 memory.go:184] no items to output this cycle I1121 15:56:46.412173 3798194 cpu.go:275] no items to output this cycle E1121 15:56:56.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:56:56.411959 3798194 memory.go:191] Add success. W1121 15:56:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:56:56.412008 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:56:56.412011 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:56:56.412357 3798194 cpu.go:282] Add success. I1121 15:56:56.441920 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:56:56.442510 3798194 net.go:642] Add success. I1121 15:56:56.446975 3798194 net.go:778] primary dev: ETH0 I1121 15:56:56.446991 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:56:56.447005 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:56:57.451022 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:56:57.451270 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:56:57.451282 3798194 disk_worker.go:723] disk space is not compliant W1121 15:56:57.451285 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:56:57.463835 3798194 disk_worker.go:509] system disk:vda1 I1121 15:56:57.463891 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:56:58.006438 3798194 disk_info.go:125] begin check local disk info of client I1121 15:56:58.010257 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:56:58.010266 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037ce80 0xc00037cec0] I1121 15:56:58.446010 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:56:59.450127 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:56:59.450208 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:56:59.450236 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:56:59.470701 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:57:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:57:06.411986 3798194 memory.go:184] no items to output this cycle I1121 15:57:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 15:57:16.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:57:16.411986 3798194 memory.go:184] no items to output this cycle I1121 15:57:16.412143 3798194 cpu.go:275] no items to output this cycle I1121 15:57:17.503534 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 15:57:17.503543 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 15:57:26.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:57:26.413274 3798194 memory.go:191] Add success. I1121 15:57:26.417685 3798194 cpu.go:282] Add success. I1121 15:57:26.441277 3798194 net.go:642] Add success. I1121 15:57:26.442882 3798194 net.go:778] primary dev: ETH0 I1121 15:57:26.442897 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:57:26.442910 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:57:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:57:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:57:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:57:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:57:36.411979 3798194 memory.go:184] no items to output this cycle I1121 15:57:36.412262 3798194 cpu.go:275] no items to output this cycle E1121 15:57:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:57:46.411981 3798194 memory.go:184] no items to output this cycle I1121 15:57:46.412137 3798194 cpu.go:275] no items to output this cycle E1121 15:57:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:57:56.411957 3798194 memory.go:191] Add success. W1121 15:57:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:57:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:57:56.412014 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:57:56.412378 3798194 cpu.go:282] Add success. I1121 15:57:56.441808 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:57:56.442352 3798194 net.go:642] Add success. I1121 15:57:56.447195 3798194 net.go:778] primary dev: ETH0 I1121 15:57:56.447211 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:57:56.447224 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:57:57.450326 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:57:57.450549 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:57:57.450561 3798194 disk_worker.go:723] disk space is not compliant W1121 15:57:57.450564 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:57:57.458688 3798194 disk_worker.go:509] system disk:vda1 I1121 15:57:57.458732 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:57:58.010388 3798194 disk_info.go:125] begin check local disk info of client I1121 15:57:58.013930 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:57:58.013940 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037ca80 0xc00037cac0] I1121 15:57:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:57:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:57:59.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:57:59.450502 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:57:59.471010 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:58:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:58:06.412255 3798194 memory.go:184] no items to output this cycle I1121 15:58:06.412400 3798194 cpu.go:275] no items to output this cycle E1121 15:58:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:58:16.411985 3798194 memory.go:184] no items to output this cycle I1121 15:58:16.412141 3798194 cpu.go:275] no items to output this cycle E1121 15:58:26.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:58:26.411948 3798194 memory.go:191] Add success. I1121 15:58:26.412281 3798194 cpu.go:282] Add success. I1121 15:58:26.441751 3798194 net.go:642] Add success. I1121 15:58:26.445887 3798194 net.go:778] primary dev: ETH0 I1121 15:58:26.445903 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:58:26.445919 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:58:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:58:29.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:58:29.450491 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:58:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:58:36.412257 3798194 memory.go:184] no items to output this cycle I1121 15:58:36.412399 3798194 cpu.go:275] no items to output this cycle E1121 15:58:46.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:58:46.412881 3798194 memory.go:184] no items to output this cycle I1121 15:58:46.413029 3798194 cpu.go:275] no items to output this cycle E1121 15:58:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:58:56.411955 3798194 memory.go:191] Add success. W1121 15:58:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:58:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:58:56.412008 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:58:56.412360 3798194 cpu.go:282] Add success. I1121 15:58:56.441816 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:58:56.442328 3798194 net.go:642] Add success. I1121 15:58:56.446711 3798194 net.go:778] primary dev: ETH0 I1121 15:58:56.446726 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:58:56.446740 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:58:57.450346 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:58:57.450569 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:58:57.450580 3798194 disk_worker.go:723] disk space is not compliant W1121 15:58:57.450583 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:58:57.464645 3798194 disk_worker.go:509] system disk:vda1 I1121 15:58:57.464694 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:58:58.014044 3798194 disk_info.go:125] begin check local disk info of client I1121 15:58:58.017697 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:58:58.017705 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047dc80 0xc00047dcc0] I1121 15:58:58.445982 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:58:59.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:58:59.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:58:59.450513 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:58:59.471027 3798194 disk_local_worker.go:436] Get disk info: [] E1121 15:59:06.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:59:06.412874 3798194 memory.go:184] no items to output this cycle I1121 15:59:06.413027 3798194 cpu.go:275] no items to output this cycle E1121 15:59:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:59:16.411980 3798194 memory.go:184] no items to output this cycle I1121 15:59:16.412136 3798194 cpu.go:275] no items to output this cycle E1121 15:59:26.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:59:26.411962 3798194 memory.go:191] Add success. I1121 15:59:26.412316 3798194 cpu.go:282] Add success. I1121 15:59:26.441979 3798194 net.go:642] Add success. I1121 15:59:26.446101 3798194 net.go:778] primary dev: ETH0 I1121 15:59:26.446129 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:59:26.446143 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:59:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:59:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:59:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 15:59:36.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:59:36.412263 3798194 memory.go:184] no items to output this cycle I1121 15:59:36.412407 3798194 cpu.go:275] no items to output this cycle E1121 15:59:46.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:59:46.412254 3798194 memory.go:184] no items to output this cycle I1121 15:59:46.412400 3798194 cpu.go:275] no items to output this cycle E1121 15:59:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 15:59:56.411948 3798194 memory.go:191] Add success. W1121 15:59:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 15:59:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 15:59:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 15:59:56.412346 3798194 cpu.go:282] Add success. I1121 15:59:56.441819 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 15:59:56.442388 3798194 net.go:642] Add success. I1121 15:59:56.446292 3798194 net.go:778] primary dev: ETH0 I1121 15:59:56.446307 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 15:59:56.446319 3798194 net.go:659] Add eni metrics success, count: 7 I1121 15:59:57.450243 3798194 custom_config.go:106] query custom config with name: gpu W1121 15:59:57.450445 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 15:59:57.450455 3798194 disk_worker.go:723] disk space is not compliant W1121 15:59:57.450459 3798194 disk_worker.go:743] disk inode is not compliant I1121 15:59:57.458499 3798194 disk_worker.go:509] system disk:vda1 I1121 15:59:57.458543 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 15:59:58.017802 3798194 disk_info.go:125] begin check local disk info of client I1121 15:59:58.021341 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 15:59:58.021349 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003faf00 0xc0003faf40] I1121 15:59:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 15:59:59.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 15:59:59.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 15:59:59.450508 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 15:59:59.471037 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:00:06.412179 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:00:06.412201 3798194 memory.go:184] no items to output this cycle I1121 16:00:06.412344 3798194 cpu.go:275] no items to output this cycle E1121 16:00:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:00:16.411982 3798194 memory.go:184] no items to output this cycle I1121 16:00:16.412141 3798194 cpu.go:275] no items to output this cycle I1121 16:00:17.504370 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:00:17.504378 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:00:26.412251 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:00:26.412290 3798194 memory.go:191] Add success. I1121 16:00:26.416417 3798194 cpu.go:282] Add success. I1121 16:00:26.441951 3798194 net.go:642] Add success. I1121 16:00:26.443615 3798194 net.go:778] primary dev: ETH0 I1121 16:00:26.443630 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:00:26.443643 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:00:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:00:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:00:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:00:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:00:36.412252 3798194 memory.go:184] no items to output this cycle I1121 16:00:36.412393 3798194 cpu.go:275] no items to output this cycle E1121 16:00:46.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:00:46.412259 3798194 memory.go:184] no items to output this cycle I1121 16:00:46.412404 3798194 cpu.go:275] no items to output this cycle E1121 16:00:56.411910 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:00:56.411942 3798194 memory.go:191] Add success. W1121 16:00:56.411968 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:00:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:00:56.411983 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:00:56.412329 3798194 cpu.go:282] Add success. I1121 16:00:56.444462 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:00:56.444949 3798194 net.go:642] Add success. I1121 16:00:56.448968 3798194 net.go:778] primary dev: ETH0 I1121 16:00:56.448983 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:00:56.448995 3798194 net.go:659] Add eni metrics success, count: 7 E1121 16:00:57.451087 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 16:00:57.451101 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:00:57.451287 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:00:57.451338 3798194 disk_worker.go:723] disk space is not compliant W1121 16:00:57.451341 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:00:57.459770 3798194 disk_worker.go:509] system disk:vda1 I1121 16:00:57.459811 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:00:58.021451 3798194 disk_info.go:125] begin check local disk info of client I1121 16:00:58.025326 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:00:58.025335 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003073c0 0xc000307400] E1121 16:00:58.448079 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 16:00:58.448094 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 16:00:59.448623 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 16:00:59.449697 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:00:59.449745 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:00:59.449764 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:00:59.471283 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:01:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:01:06.412256 3798194 memory.go:184] no items to output this cycle I1121 16:01:06.412397 3798194 cpu.go:275] no items to output this cycle E1121 16:01:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:01:16.411981 3798194 memory.go:184] no items to output this cycle I1121 16:01:16.412138 3798194 cpu.go:275] no items to output this cycle E1121 16:01:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:01:26.411952 3798194 memory.go:191] Add success. I1121 16:01:26.412298 3798194 cpu.go:282] Add success. I1121 16:01:26.442086 3798194 net.go:642] Add success. I1121 16:01:26.446530 3798194 net.go:778] primary dev: ETH0 I1121 16:01:26.446545 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:01:26.446560 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:01:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:01:29.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:01:29.450148 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:01:36.411945 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:01:36.411967 3798194 memory.go:184] no items to output this cycle I1121 16:01:36.412126 3798194 cpu.go:275] no items to output this cycle E1121 16:01:46.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:01:46.411990 3798194 memory.go:184] no items to output this cycle I1121 16:01:46.412270 3798194 cpu.go:275] no items to output this cycle E1121 16:01:56.411936 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:01:56.411972 3798194 memory.go:191] Add success. W1121 16:01:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:01:56.412015 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:01:56.412019 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:01:56.412397 3798194 cpu.go:282] Add success. I1121 16:01:56.440964 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:01:56.441476 3798194 net.go:642] Add success. I1121 16:01:56.445784 3798194 net.go:778] primary dev: ETH0 I1121 16:01:56.445799 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:01:56.445812 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:01:57.445284 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:01:57.445558 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:01:57.445569 3798194 disk_worker.go:723] disk space is not compliant W1121 16:01:57.445572 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:01:57.448037 3798194 disk_worker.go:509] system disk:vda1 I1121 16:01:57.448080 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:01:58.025441 3798194 disk_info.go:125] begin check local disk info of client I1121 16:01:58.029053 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:01:58.029062 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e4e40 0xc0002e4e80] I1121 16:01:58.445983 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:01:59.450030 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:01:59.450128 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:01:59.450159 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:01:59.471653 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:02:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:02:06.411980 3798194 memory.go:184] no items to output this cycle I1121 16:02:06.412140 3798194 cpu.go:275] no items to output this cycle E1121 16:02:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:02:16.411975 3798194 memory.go:184] no items to output this cycle I1121 16:02:16.412135 3798194 cpu.go:275] no items to output this cycle E1121 16:02:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:02:26.411953 3798194 memory.go:191] Add success. I1121 16:02:26.412310 3798194 cpu.go:282] Add success. I1121 16:02:26.442040 3798194 net.go:642] Add success. I1121 16:02:26.445982 3798194 net.go:778] primary dev: ETH0 I1121 16:02:26.445996 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:02:26.446009 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:02:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:02:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:02:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:02:36.411253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:02:36.411276 3798194 memory.go:184] no items to output this cycle I1121 16:02:36.411417 3798194 cpu.go:275] no items to output this cycle E1121 16:02:46.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:02:46.411989 3798194 memory.go:184] no items to output this cycle I1121 16:02:46.412149 3798194 cpu.go:275] no items to output this cycle E1121 16:02:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:02:56.411953 3798194 memory.go:191] Add success. W1121 16:02:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:02:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:02:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:02:56.412360 3798194 cpu.go:282] Add success. I1121 16:02:56.441821 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:02:56.442448 3798194 net.go:642] Add success. I1121 16:02:56.446594 3798194 net.go:778] primary dev: ETH0 I1121 16:02:56.446609 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:02:56.446625 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:02:57.450663 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:02:57.450818 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:02:57.450868 3798194 disk_worker.go:723] disk space is not compliant W1121 16:02:57.450871 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:02:57.459343 3798194 disk_worker.go:509] system disk:vda1 I1121 16:02:57.459410 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:02:58.029168 3798194 disk_info.go:125] begin check local disk info of client I1121 16:02:58.033073 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:02:58.033083 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00048b040 0xc00048b080] I1121 16:02:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:02:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:02:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:02:59.450149 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:02:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:03:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:03:06.412257 3798194 memory.go:184] no items to output this cycle I1121 16:03:06.412399 3798194 cpu.go:275] no items to output this cycle E1121 16:03:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:03:16.411978 3798194 memory.go:184] no items to output this cycle I1121 16:03:16.412144 3798194 cpu.go:275] no items to output this cycle I1121 16:03:17.505374 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:03:17.505383 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:03:26.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:03:26.412271 3798194 memory.go:191] Add success. I1121 16:03:26.416486 3798194 cpu.go:282] Add success. I1121 16:03:26.442076 3798194 net.go:642] Add success. I1121 16:03:26.443669 3798194 net.go:778] primary dev: ETH0 I1121 16:03:26.443685 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:03:26.443698 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:03:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:03:29.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:03:29.450493 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:03:36.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:03:36.411985 3798194 memory.go:184] no items to output this cycle I1121 16:03:36.412138 3798194 cpu.go:275] no items to output this cycle E1121 16:03:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:03:46.411973 3798194 memory.go:184] no items to output this cycle I1121 16:03:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 16:03:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:03:56.411954 3798194 memory.go:191] Add success. W1121 16:03:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:03:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:03:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:03:56.412344 3798194 cpu.go:282] Add success. I1121 16:03:56.441820 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:03:56.442419 3798194 net.go:642] Add success. I1121 16:03:56.446716 3798194 net.go:778] primary dev: ETH0 I1121 16:03:56.446732 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:03:56.446742 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:03:57.450260 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:03:57.450457 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:03:57.450467 3798194 disk_worker.go:723] disk space is not compliant W1121 16:03:57.450470 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:03:57.458498 3798194 disk_worker.go:509] system disk:vda1 I1121 16:03:57.458540 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:03:58.033176 3798194 disk_info.go:125] begin check local disk info of client I1121 16:03:58.036674 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:03:58.036682 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b4100 0xc0003b4140] I1121 16:03:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:03:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:03:59.450127 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:03:59.450158 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:03:59.471694 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:04:06.412086 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:04:06.412122 3798194 memory.go:184] no items to output this cycle I1121 16:04:06.412277 3798194 cpu.go:275] no items to output this cycle E1121 16:04:16.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:04:16.412255 3798194 memory.go:184] no items to output this cycle I1121 16:04:16.412405 3798194 cpu.go:275] no items to output this cycle E1121 16:04:26.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:04:26.412273 3798194 memory.go:191] Add success. I1121 16:04:26.416510 3798194 cpu.go:282] Add success. I1121 16:04:26.442397 3798194 net.go:642] Add success. I1121 16:04:26.443851 3798194 net.go:778] primary dev: ETH0 I1121 16:04:26.443866 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:04:26.443881 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:04:29.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:04:29.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:04:29.450152 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:04:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:04:36.411981 3798194 memory.go:184] no items to output this cycle I1121 16:04:36.412136 3798194 cpu.go:275] no items to output this cycle E1121 16:04:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:04:46.411979 3798194 memory.go:184] no items to output this cycle I1121 16:04:46.412134 3798194 cpu.go:275] no items to output this cycle E1121 16:04:56.412246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:04:56.412280 3798194 memory.go:191] Add success. W1121 16:04:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:04:56.412316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:04:56.412320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:04:56.418523 3798194 cpu.go:282] Add success. I1121 16:04:56.440918 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:04:56.441487 3798194 net.go:642] Add success. I1121 16:04:56.443075 3798194 net.go:778] primary dev: ETH0 I1121 16:04:56.443091 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:04:56.443105 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:04:57.445321 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:04:57.445575 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:04:57.445588 3798194 disk_worker.go:723] disk space is not compliant W1121 16:04:57.445591 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:04:57.448074 3798194 disk_worker.go:509] system disk:vda1 I1121 16:04:57.448136 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:04:58.036786 3798194 disk_info.go:125] begin check local disk info of client I1121 16:04:58.040345 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:04:58.040354 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00025d500 0xc00025d540] I1121 16:04:58.445987 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:04:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:04:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:04:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:04:59.471646 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:05:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:05:06.411978 3798194 memory.go:184] no items to output this cycle I1121 16:05:06.412139 3798194 cpu.go:275] no items to output this cycle E1121 16:05:16.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:05:16.412251 3798194 memory.go:184] no items to output this cycle I1121 16:05:16.412395 3798194 cpu.go:275] no items to output this cycle E1121 16:05:26.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:05:26.411961 3798194 memory.go:191] Add success. I1121 16:05:26.412309 3798194 cpu.go:282] Add success. I1121 16:05:26.441931 3798194 net.go:642] Add success. I1121 16:05:26.446207 3798194 net.go:778] primary dev: ETH0 I1121 16:05:26.446227 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:05:26.446240 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:05:29.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:05:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:05:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:05:36.412097 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:05:36.412136 3798194 memory.go:184] no items to output this cycle I1121 16:05:36.412282 3798194 cpu.go:275] no items to output this cycle E1121 16:05:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:05:46.411977 3798194 memory.go:184] no items to output this cycle I1121 16:05:46.412133 3798194 cpu.go:275] no items to output this cycle E1121 16:05:56.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:05:56.412887 3798194 memory.go:191] Add success. W1121 16:05:56.412920 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:05:56.412931 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:05:56.412935 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:05:56.416999 3798194 cpu.go:282] Add success. I1121 16:05:56.444086 3798194 event_worker.go:152] Polling the log file for events... I1121 16:05:56.444457 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:05:56.445393 3798194 net.go:642] Add success. I1121 16:05:56.446877 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795246556, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 16:05:56.447172 3798194 net.go:778] primary dev: ETH0 I1121 16:05:56.447187 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:05:56.452548 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:05:56.465193 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 16:05:57.449221 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 16:05:57.449235 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:05:57.449465 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:05:57.449476 3798194 disk_worker.go:723] disk space is not compliant W1121 16:05:57.449479 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:05:57.451276 3798194 disk_worker.go:509] system disk:vda1 I1121 16:05:57.451317 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:05:58.040456 3798194 disk_info.go:125] begin check local disk info of client I1121 16:05:58.044346 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:05:58.044355 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004ba440 0xc0004ba480] E1121 16:05:58.448514 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 16:05:58.448529 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 16:05:59.448971 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 16:05:59.450045 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:05:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:05:59.450120 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:05:59.471623 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:06:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:06:06.411982 3798194 memory.go:184] no items to output this cycle I1121 16:06:06.412168 3798194 cpu.go:275] no items to output this cycle E1121 16:06:16.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:06:16.413270 3798194 memory.go:184] no items to output this cycle I1121 16:06:16.413449 3798194 cpu.go:275] no items to output this cycle I1121 16:06:17.505538 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:06:17.505546 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:06:26.411447 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:06:26.411483 3798194 memory.go:191] Add success. I1121 16:06:26.411796 3798194 cpu.go:282] Add success. I1121 16:06:26.441595 3798194 net.go:642] Add success. I1121 16:06:26.445461 3798194 net.go:778] primary dev: ETH0 I1121 16:06:26.445477 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:06:26.445491 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:06:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:06:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:06:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:06:36.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:06:36.411987 3798194 memory.go:184] no items to output this cycle I1121 16:06:36.412142 3798194 cpu.go:275] no items to output this cycle E1121 16:06:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:06:46.411979 3798194 memory.go:184] no items to output this cycle I1121 16:06:46.412140 3798194 cpu.go:275] no items to output this cycle E1121 16:06:56.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:06:56.411959 3798194 memory.go:191] Add success. W1121 16:06:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:06:56.412006 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:06:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:06:56.412375 3798194 cpu.go:282] Add success. I1121 16:06:56.440970 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:06:56.441483 3798194 net.go:642] Add success. I1121 16:06:56.445504 3798194 net.go:778] primary dev: ETH0 I1121 16:06:56.445518 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:06:56.445531 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:06:57.450665 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:06:57.450825 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:06:57.450880 3798194 disk_worker.go:723] disk space is not compliant W1121 16:06:57.450883 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:06:57.458943 3798194 disk_worker.go:509] system disk:vda1 I1121 16:06:57.458982 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:06:58.044460 3798194 disk_info.go:125] begin check local disk info of client I1121 16:06:58.048351 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:06:58.048360 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030fac0 0xc00030fb00] I1121 16:06:58.446012 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:06:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:06:59.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:06:59.450500 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:06:59.471008 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:07:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:07:06.412254 3798194 memory.go:184] no items to output this cycle I1121 16:07:06.412394 3798194 cpu.go:275] no items to output this cycle E1121 16:07:16.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:07:16.413250 3798194 memory.go:184] no items to output this cycle I1121 16:07:16.413394 3798194 cpu.go:275] no items to output this cycle E1121 16:07:26.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:07:26.412287 3798194 memory.go:191] Add success. I1121 16:07:26.417089 3798194 cpu.go:282] Add success. I1121 16:07:26.441632 3798194 net.go:642] Add success. I1121 16:07:26.443180 3798194 net.go:778] primary dev: ETH0 I1121 16:07:26.443194 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:07:26.443207 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:07:29.450314 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:07:29.450394 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:07:29.450420 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:07:36.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:07:36.411983 3798194 memory.go:184] no items to output this cycle I1121 16:07:36.412139 3798194 cpu.go:275] no items to output this cycle E1121 16:07:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:07:46.411973 3798194 memory.go:184] no items to output this cycle I1121 16:07:46.412140 3798194 cpu.go:275] no items to output this cycle E1121 16:07:56.413532 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:07:56.413569 3798194 memory.go:191] Add success. W1121 16:07:56.413602 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:07:56.413614 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:07:56.413618 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:07:56.418402 3798194 cpu.go:282] Add success. I1121 16:07:56.441058 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:07:56.441781 3798194 net.go:642] Add success. I1121 16:07:56.443953 3798194 net.go:778] primary dev: ETH0 I1121 16:07:56.443968 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:07:56.443981 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:07:57.452350 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:07:57.452528 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:07:57.452584 3798194 disk_worker.go:723] disk space is not compliant W1121 16:07:57.452587 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:07:57.463921 3798194 disk_worker.go:509] system disk:vda1 I1121 16:07:57.463965 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:07:58.048467 3798194 disk_info.go:125] begin check local disk info of client I1121 16:07:58.052006 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:07:58.052014 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000268d00 0xc000268d40] I1121 16:07:58.446400 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:07:59.450499 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:07:59.450585 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:07:59.450615 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:07:59.471126 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:08:06.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:08:06.413264 3798194 memory.go:184] no items to output this cycle I1121 16:08:06.413405 3798194 cpu.go:275] no items to output this cycle E1121 16:08:16.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:08:16.413258 3798194 memory.go:184] no items to output this cycle I1121 16:08:16.413419 3798194 cpu.go:275] no items to output this cycle E1121 16:08:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:08:26.413264 3798194 memory.go:191] Add success. I1121 16:08:26.417619 3798194 cpu.go:282] Add success. I1121 16:08:26.442164 3798194 net.go:642] Add success. I1121 16:08:26.443686 3798194 net.go:778] primary dev: ETH0 I1121 16:08:26.443710 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:08:26.443724 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:08:29.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:08:29.450462 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:08:29.450488 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:08:36.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:08:36.413255 3798194 memory.go:184] no items to output this cycle I1121 16:08:36.413401 3798194 cpu.go:275] no items to output this cycle E1121 16:08:46.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:08:46.412257 3798194 memory.go:184] no items to output this cycle I1121 16:08:46.412403 3798194 cpu.go:275] no items to output this cycle E1121 16:08:56.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:08:56.413271 3798194 memory.go:191] Add success. W1121 16:08:56.413299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:08:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:08:56.413315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:08:56.417756 3798194 cpu.go:282] Add success. I1121 16:08:56.441126 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:08:56.441931 3798194 net.go:642] Add success. I1121 16:08:56.451816 3798194 net.go:778] primary dev: ETH0 I1121 16:08:56.451833 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:08:56.451847 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:08:57.451143 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:08:57.451346 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:08:57.451359 3798194 disk_worker.go:723] disk space is not compliant W1121 16:08:57.451362 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:08:57.459731 3798194 disk_worker.go:509] system disk:vda1 I1121 16:08:57.459774 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:08:58.052128 3798194 disk_info.go:125] begin check local disk info of client I1121 16:08:58.055969 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:08:58.055978 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040e980 0xc00040e9c0] I1121 16:08:58.446390 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:08:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:08:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:08:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:08:59.471667 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:09:06.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:09:06.412876 3798194 memory.go:184] no items to output this cycle I1121 16:09:06.413021 3798194 cpu.go:275] no items to output this cycle E1121 16:09:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:09:16.411977 3798194 memory.go:184] no items to output this cycle I1121 16:09:16.412138 3798194 cpu.go:275] no items to output this cycle I1121 16:09:17.506384 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:09:17.506392 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:09:26.411498 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:09:26.411531 3798194 memory.go:191] Add success. I1121 16:09:26.411861 3798194 cpu.go:282] Add success. I1121 16:09:26.441508 3798194 net.go:642] Add success. I1121 16:09:26.445258 3798194 net.go:778] primary dev: ETH0 I1121 16:09:26.445273 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:09:26.445286 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:09:29.450454 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:09:29.450535 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:09:29.450561 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:09:36.412528 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:09:36.412560 3798194 memory.go:184] no items to output this cycle I1121 16:09:36.412705 3798194 cpu.go:275] no items to output this cycle E1121 16:09:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:09:46.411987 3798194 memory.go:184] no items to output this cycle I1121 16:09:46.412150 3798194 cpu.go:275] no items to output this cycle E1121 16:09:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:09:56.411944 3798194 memory.go:191] Add success. W1121 16:09:56.411970 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:09:56.411981 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:09:56.411984 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:09:56.412339 3798194 cpu.go:282] Add success. I1121 16:09:56.441838 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:09:56.442649 3798194 net.go:642] Add success. I1121 16:09:56.446441 3798194 net.go:778] primary dev: ETH0 I1121 16:09:56.446458 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:09:56.446471 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:09:57.450314 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:09:57.450470 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:09:57.450522 3798194 disk_worker.go:723] disk space is not compliant W1121 16:09:57.450525 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:09:57.458557 3798194 disk_worker.go:509] system disk:vda1 I1121 16:09:57.458599 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:09:58.056081 3798194 disk_info.go:125] begin check local disk info of client I1121 16:09:58.059897 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:09:58.059906 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ac840 0xc0003ac880] I1121 16:09:58.446009 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:09:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:09:59.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:09:59.450510 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:09:59.471034 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:10:06.412763 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:10:06.412786 3798194 memory.go:184] no items to output this cycle I1121 16:10:06.412939 3798194 cpu.go:275] no items to output this cycle E1121 16:10:16.412254 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:10:16.412277 3798194 memory.go:184] no items to output this cycle I1121 16:10:16.412432 3798194 cpu.go:275] no items to output this cycle E1121 16:10:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:10:26.411958 3798194 memory.go:191] Add success. I1121 16:10:26.412319 3798194 cpu.go:282] Add success. I1121 16:10:26.441919 3798194 net.go:642] Add success. I1121 16:10:26.446048 3798194 net.go:778] primary dev: ETH0 I1121 16:10:26.446065 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:10:26.446078 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:10:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:10:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:10:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:10:36.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:10:36.412254 3798194 memory.go:184] no items to output this cycle I1121 16:10:36.412399 3798194 cpu.go:275] no items to output this cycle E1121 16:10:46.411556 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:10:46.411579 3798194 memory.go:184] no items to output this cycle I1121 16:10:46.411723 3798194 cpu.go:275] no items to output this cycle E1121 16:10:56.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:10:56.413266 3798194 memory.go:191] Add success. W1121 16:10:56.413293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:10:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:10:56.413308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:10:56.417616 3798194 cpu.go:282] Add success. I1121 16:10:56.444212 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:10:56.444835 3798194 net.go:642] Add success. I1121 16:10:56.446360 3798194 net.go:778] primary dev: ETH0 I1121 16:10:56.446381 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:10:56.446395 3798194 net.go:659] Add eni metrics success, count: 7 E1121 16:10:57.450937 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 16:10:57.450951 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:10:57.451098 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:10:57.451173 3798194 disk_worker.go:723] disk space is not compliant W1121 16:10:57.451177 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:10:57.459988 3798194 disk_worker.go:509] system disk:vda1 I1121 16:10:57.460034 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:10:58.060010 3798194 disk_info.go:125] begin check local disk info of client I1121 16:10:58.063809 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:10:58.063818 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00042e880 0xc00042e8c0] E1121 16:10:58.448097 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 16:10:58.448128 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 16:10:59.448631 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 16:10:59.449706 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:10:59.449756 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:10:59.449775 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:10:59.471286 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:11:06.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:11:06.412877 3798194 memory.go:184] no items to output this cycle I1121 16:11:06.413022 3798194 cpu.go:275] no items to output this cycle E1121 16:11:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:11:16.411973 3798194 memory.go:184] no items to output this cycle I1121 16:11:16.412139 3798194 cpu.go:275] no items to output this cycle E1121 16:11:26.412902 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:11:26.412936 3798194 memory.go:191] Add success. I1121 16:11:26.417240 3798194 cpu.go:282] Add success. I1121 16:11:26.442049 3798194 net.go:642] Add success. I1121 16:11:26.443628 3798194 net.go:778] primary dev: ETH0 I1121 16:11:26.443642 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:11:26.443654 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:11:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:11:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:11:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:11:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:11:36.411981 3798194 memory.go:184] no items to output this cycle I1121 16:11:36.412140 3798194 cpu.go:275] no items to output this cycle E1121 16:11:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:11:46.411977 3798194 memory.go:184] no items to output this cycle I1121 16:11:46.412135 3798194 cpu.go:275] no items to output this cycle E1121 16:11:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:11:56.411957 3798194 memory.go:191] Add success. W1121 16:11:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:11:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:11:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:11:56.412363 3798194 cpu.go:282] Add success. I1121 16:11:56.441819 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:11:56.442510 3798194 net.go:642] Add success. I1121 16:11:56.446570 3798194 net.go:778] primary dev: ETH0 I1121 16:11:56.446588 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:11:56.446606 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:11:57.450345 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:11:57.450548 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:11:57.450558 3798194 disk_worker.go:723] disk space is not compliant W1121 16:11:57.450561 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:11:57.458692 3798194 disk_worker.go:509] system disk:vda1 I1121 16:11:57.458733 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:11:58.063920 3798194 disk_info.go:125] begin check local disk info of client I1121 16:11:58.067453 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:11:58.067462 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003aa100 0xc0003aa140] I1121 16:11:58.446010 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:11:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:11:59.450122 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:11:59.450151 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:11:59.471669 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:12:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:12:06.412254 3798194 memory.go:184] no items to output this cycle I1121 16:12:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 16:12:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:12:16.411976 3798194 memory.go:184] no items to output this cycle I1121 16:12:16.412141 3798194 cpu.go:275] no items to output this cycle I1121 16:12:17.506561 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:12:17.506569 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:12:26.411329 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:12:26.411362 3798194 memory.go:191] Add success. I1121 16:12:26.411708 3798194 cpu.go:282] Add success. I1121 16:12:26.441287 3798194 net.go:642] Add success. I1121 16:12:26.445776 3798194 net.go:778] primary dev: ETH0 I1121 16:12:26.445792 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:12:26.445804 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:12:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:12:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:12:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:12:36.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:12:36.413251 3798194 memory.go:184] no items to output this cycle I1121 16:12:36.413398 3798194 cpu.go:275] no items to output this cycle E1121 16:12:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:12:46.412261 3798194 memory.go:184] no items to output this cycle I1121 16:12:46.412404 3798194 cpu.go:275] no items to output this cycle E1121 16:12:56.411931 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:12:56.411964 3798194 memory.go:191] Add success. W1121 16:12:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:12:56.412003 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:12:56.412007 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:12:56.412344 3798194 cpu.go:282] Add success. I1121 16:12:56.441860 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:12:56.442372 3798194 net.go:642] Add success. I1121 16:12:56.447255 3798194 net.go:778] primary dev: ETH0 I1121 16:12:56.447271 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:12:56.447283 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:12:57.450247 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:12:57.450462 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:12:57.450473 3798194 disk_worker.go:723] disk space is not compliant W1121 16:12:57.450476 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:12:57.458774 3798194 disk_worker.go:509] system disk:vda1 I1121 16:12:57.458818 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:12:58.069136 3798194 disk_info.go:125] begin check local disk info of client I1121 16:12:58.072561 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:12:58.072570 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00039dec0 0xc00039df00] I1121 16:12:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:12:59.450030 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:12:59.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:12:59.450151 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:12:59.471683 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:13:06.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:13:06.412882 3798194 memory.go:184] no items to output this cycle I1121 16:13:06.413050 3798194 cpu.go:275] no items to output this cycle E1121 16:13:16.412134 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:13:16.412165 3798194 memory.go:184] no items to output this cycle I1121 16:13:16.412469 3798194 cpu.go:275] no items to output this cycle E1121 16:13:26.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:13:26.412275 3798194 memory.go:191] Add success. I1121 16:13:26.417043 3798194 cpu.go:282] Add success. I1121 16:13:26.441599 3798194 net.go:642] Add success. I1121 16:13:26.443145 3798194 net.go:778] primary dev: ETH0 I1121 16:13:26.443159 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:13:26.443173 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:13:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:13:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:13:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:13:36.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:13:36.413251 3798194 memory.go:184] no items to output this cycle I1121 16:13:36.413397 3798194 cpu.go:275] no items to output this cycle E1121 16:13:46.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:13:46.411997 3798194 memory.go:184] no items to output this cycle I1121 16:13:46.412163 3798194 cpu.go:275] no items to output this cycle E1121 16:13:56.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:13:56.412274 3798194 memory.go:191] Add success. W1121 16:13:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:13:56.412322 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:13:56.412325 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:13:56.416451 3798194 cpu.go:282] Add success. I1121 16:13:56.441766 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:13:56.442270 3798194 net.go:642] Add success. I1121 16:13:56.443919 3798194 net.go:778] primary dev: ETH0 I1121 16:13:56.443934 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:13:56.443946 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:13:57.450430 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:13:57.450590 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:13:57.450643 3798194 disk_worker.go:723] disk space is not compliant W1121 16:13:57.450646 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:13:57.458750 3798194 disk_worker.go:509] system disk:vda1 I1121 16:13:57.458801 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:13:58.074144 3798194 disk_info.go:125] begin check local disk info of client I1121 16:13:58.077659 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:13:58.077667 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000313f00 0xc000313f40] I1121 16:13:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:13:59.450107 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:13:59.450204 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:13:59.450233 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:13:59.470708 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:14:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:14:06.411981 3798194 memory.go:184] no items to output this cycle I1121 16:14:06.412133 3798194 cpu.go:275] no items to output this cycle E1121 16:14:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:14:16.411978 3798194 memory.go:184] no items to output this cycle I1121 16:14:16.412143 3798194 cpu.go:275] no items to output this cycle E1121 16:14:26.412437 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:14:26.412474 3798194 memory.go:191] Add success. I1121 16:14:26.416669 3798194 cpu.go:282] Add success. I1121 16:14:26.442186 3798194 net.go:642] Add success. I1121 16:14:26.443542 3798194 net.go:778] primary dev: ETH0 I1121 16:14:26.443557 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:14:26.443571 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:14:29.450003 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:14:29.450088 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:14:29.450125 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:14:36.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:14:36.411985 3798194 memory.go:184] no items to output this cycle I1121 16:14:36.412141 3798194 cpu.go:275] no items to output this cycle E1121 16:14:46.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:14:46.412265 3798194 memory.go:184] no items to output this cycle I1121 16:14:46.412544 3798194 cpu.go:275] no items to output this cycle E1121 16:14:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:14:56.411954 3798194 memory.go:191] Add success. W1121 16:14:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:14:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:14:56.412000 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:14:56.412367 3798194 cpu.go:282] Add success. I1121 16:14:56.441819 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:14:56.442433 3798194 net.go:642] Add success. I1121 16:14:56.446300 3798194 net.go:778] primary dev: ETH0 I1121 16:14:56.446316 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:14:56.446330 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:14:57.450231 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:14:57.450404 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:14:57.450459 3798194 disk_worker.go:723] disk space is not compliant W1121 16:14:57.450462 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:14:57.461049 3798194 disk_worker.go:509] system disk:vda1 I1121 16:14:57.461091 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:14:58.077767 3798194 disk_info.go:125] begin check local disk info of client I1121 16:14:58.081284 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:14:58.081293 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000444100 0xc000444140] I1121 16:14:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:14:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:14:59.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:14:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:14:59.471664 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:15:06.412087 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:15:06.412128 3798194 memory.go:184] no items to output this cycle I1121 16:15:06.412273 3798194 cpu.go:275] no items to output this cycle E1121 16:15:16.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:15:16.413255 3798194 memory.go:184] no items to output this cycle I1121 16:15:16.413401 3798194 cpu.go:275] no items to output this cycle I1121 16:15:17.506721 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:15:17.506729 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:15:26.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:15:26.412274 3798194 memory.go:191] Add success. I1121 16:15:26.416356 3798194 cpu.go:282] Add success. I1121 16:15:26.441938 3798194 net.go:642] Add success. I1121 16:15:26.443442 3798194 net.go:778] primary dev: ETH0 I1121 16:15:26.443456 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:15:26.443467 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:15:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:15:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:15:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:15:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:15:36.411981 3798194 memory.go:184] no items to output this cycle I1121 16:15:36.412138 3798194 cpu.go:275] no items to output this cycle E1121 16:15:46.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:15:46.411973 3798194 memory.go:184] no items to output this cycle I1121 16:15:46.412129 3798194 cpu.go:275] no items to output this cycle E1121 16:15:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:15:56.411948 3798194 memory.go:191] Add success. W1121 16:15:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:15:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:15:56.411990 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:15:56.412349 3798194 cpu.go:282] Add success. I1121 16:15:56.441567 3798194 event_worker.go:152] Polling the log file for events... I1121 16:15:56.444731 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:15:56.445249 3798194 net.go:642] Add success. I1121 16:15:56.449233 3798194 net.go:778] primary dev: ETH0 I1121 16:15:56.449247 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:15:56.449261 3798194 net.go:659] Add eni metrics success, count: 7 E1121 16:15:57.450864 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 16:15:57.450879 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:15:57.451038 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:15:57.451088 3798194 disk_worker.go:723] disk space is not compliant W1121 16:15:57.451091 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:15:57.459411 3798194 disk_worker.go:509] system disk:vda1 I1121 16:15:57.459454 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:15:58.081394 3798194 disk_info.go:125] begin check local disk info of client I1121 16:15:58.085206 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:15:58.085215 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047f080 0xc00047f0c0] E1121 16:15:58.448085 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 16:15:58.448099 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 16:15:59.448592 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 16:15:59.449667 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:15:59.449720 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:15:59.449739 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:15:59.471289 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:16:06.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:16:06.412250 3798194 memory.go:184] no items to output this cycle I1121 16:16:06.412394 3798194 cpu.go:275] no items to output this cycle E1121 16:16:16.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:16:16.412255 3798194 memory.go:184] no items to output this cycle I1121 16:16:16.412399 3798194 cpu.go:275] no items to output this cycle E1121 16:16:26.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:16:26.411951 3798194 memory.go:191] Add success. I1121 16:16:26.412294 3798194 cpu.go:282] Add success. I1121 16:16:26.441745 3798194 net.go:642] Add success. I1121 16:16:26.445863 3798194 net.go:778] primary dev: ETH0 I1121 16:16:26.445879 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:16:26.445892 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:16:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:16:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:16:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:16:36.413171 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:16:36.413196 3798194 memory.go:184] no items to output this cycle I1121 16:16:36.413342 3798194 cpu.go:275] no items to output this cycle E1121 16:16:46.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:16:46.413265 3798194 memory.go:184] no items to output this cycle I1121 16:16:46.413411 3798194 cpu.go:275] no items to output this cycle E1121 16:16:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:16:56.411950 3798194 memory.go:191] Add success. W1121 16:16:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:16:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:16:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:16:56.412356 3798194 cpu.go:282] Add success. I1121 16:16:56.441799 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:16:56.442387 3798194 net.go:642] Add success. I1121 16:16:56.456070 3798194 net.go:778] primary dev: ETH0 I1121 16:16:56.456089 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:16:56.456101 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:16:57.449990 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:16:57.450221 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:16:57.450232 3798194 disk_worker.go:723] disk space is not compliant W1121 16:16:57.450235 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:16:57.458271 3798194 disk_worker.go:509] system disk:vda1 I1121 16:16:57.458313 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:16:58.087135 3798194 disk_info.go:125] begin check local disk info of client I1121 16:16:58.090638 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:16:58.090647 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b8440 0xc0003b8480] I1121 16:16:58.446012 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:16:59.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:16:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:16:59.450144 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:16:59.471668 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:17:06.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:17:06.413259 3798194 memory.go:184] no items to output this cycle I1121 16:17:06.413406 3798194 cpu.go:275] no items to output this cycle E1121 16:17:16.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:17:16.413267 3798194 memory.go:184] no items to output this cycle I1121 16:17:16.413420 3798194 cpu.go:275] no items to output this cycle E1121 16:17:26.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:17:26.412265 3798194 memory.go:191] Add success. I1121 16:17:26.416463 3798194 cpu.go:282] Add success. I1121 16:17:26.442356 3798194 net.go:642] Add success. I1121 16:17:26.443953 3798194 net.go:778] primary dev: ETH0 I1121 16:17:26.443969 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:17:26.443987 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:17:29.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:17:29.450118 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:17:29.450149 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:17:36.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:17:36.411989 3798194 memory.go:184] no items to output this cycle I1121 16:17:36.412147 3798194 cpu.go:275] no items to output this cycle E1121 16:17:46.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:17:46.411991 3798194 memory.go:184] no items to output this cycle I1121 16:17:46.412164 3798194 cpu.go:275] no items to output this cycle E1121 16:17:56.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:17:56.411980 3798194 memory.go:191] Add success. W1121 16:17:56.412012 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:17:56.412025 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:17:56.412029 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:17:56.412401 3798194 cpu.go:282] Add success. I1121 16:17:56.441894 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:17:56.442541 3798194 net.go:642] Add success. I1121 16:17:56.446428 3798194 net.go:778] primary dev: ETH0 I1121 16:17:56.446443 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:17:56.446457 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:17:57.450635 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:17:57.450861 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:17:57.450873 3798194 disk_worker.go:723] disk space is not compliant W1121 16:17:57.450877 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:17:57.459143 3798194 disk_worker.go:509] system disk:vda1 I1121 16:17:57.459187 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:17:58.090751 3798194 disk_info.go:125] begin check local disk info of client I1121 16:17:58.094647 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:17:58.094656 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b8600 0xc0003b8640] I1121 16:17:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:17:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:17:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:17:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:17:59.471651 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:18:06.412710 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:18:06.412734 3798194 memory.go:184] no items to output this cycle I1121 16:18:06.412893 3798194 cpu.go:275] no items to output this cycle E1121 16:18:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:18:16.411978 3798194 memory.go:184] no items to output this cycle I1121 16:18:16.412133 3798194 cpu.go:275] no items to output this cycle I1121 16:18:17.507380 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:18:17.507389 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:18:26.411224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:18:26.411256 3798194 memory.go:191] Add success. I1121 16:18:26.411598 3798194 cpu.go:282] Add success. I1121 16:18:26.442221 3798194 net.go:642] Add success. I1121 16:18:26.446263 3798194 net.go:778] primary dev: ETH0 I1121 16:18:26.446277 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:18:26.446290 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:18:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:18:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:18:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:18:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:18:36.412252 3798194 memory.go:184] no items to output this cycle I1121 16:18:36.412389 3798194 cpu.go:275] no items to output this cycle E1121 16:18:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:18:46.412253 3798194 memory.go:184] no items to output this cycle I1121 16:18:46.412415 3798194 cpu.go:275] no items to output this cycle E1121 16:18:56.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:18:56.413272 3798194 memory.go:191] Add success. W1121 16:18:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:18:56.413316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:18:56.413319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:18:56.417539 3798194 cpu.go:282] Add success. I1121 16:18:56.442124 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:18:56.443222 3798194 net.go:642] Add success. I1121 16:18:56.444861 3798194 net.go:778] primary dev: ETH0 I1121 16:18:56.444977 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:18:56.444991 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:18:57.450307 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:18:57.450516 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:18:57.450527 3798194 disk_worker.go:723] disk space is not compliant W1121 16:18:57.450530 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:18:57.458726 3798194 disk_worker.go:509] system disk:vda1 I1121 16:18:57.458764 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:18:58.097214 3798194 disk_info.go:125] begin check local disk info of client I1121 16:18:58.100788 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:18:58.100797 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004ee1c0 0xc0004ee200] I1121 16:18:58.446390 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:18:59.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:18:59.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:18:59.450493 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:18:59.471003 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:19:06.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:19:06.412252 3798194 memory.go:184] no items to output this cycle I1121 16:19:06.412399 3798194 cpu.go:275] no items to output this cycle E1121 16:19:16.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:19:16.412885 3798194 memory.go:184] no items to output this cycle I1121 16:19:16.413031 3798194 cpu.go:275] no items to output this cycle E1121 16:19:26.411931 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:19:26.411966 3798194 memory.go:191] Add success. I1121 16:19:26.412317 3798194 cpu.go:282] Add success. I1121 16:19:26.441992 3798194 net.go:642] Add success. I1121 16:19:26.452192 3798194 net.go:778] primary dev: ETH0 I1121 16:19:26.452207 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:19:26.452220 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:19:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:19:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:19:29.450145 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:19:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:19:36.412253 3798194 memory.go:184] no items to output this cycle I1121 16:19:36.412408 3798194 cpu.go:275] no items to output this cycle E1121 16:19:46.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:19:46.411985 3798194 memory.go:184] no items to output this cycle I1121 16:19:46.412142 3798194 cpu.go:275] no items to output this cycle E1121 16:19:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:19:56.411951 3798194 memory.go:191] Add success. W1121 16:19:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:19:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:19:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:19:56.412353 3798194 cpu.go:282] Add success. I1121 16:19:56.441814 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:19:56.442344 3798194 net.go:642] Add success. I1121 16:19:56.446520 3798194 net.go:778] primary dev: ETH0 I1121 16:19:56.446534 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:19:56.446547 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:19:57.450622 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:19:57.450782 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:19:57.450837 3798194 disk_worker.go:723] disk space is not compliant W1121 16:19:57.450840 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:19:57.459340 3798194 disk_worker.go:509] system disk:vda1 I1121 16:19:57.459383 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:19:58.103136 3798194 disk_info.go:125] begin check local disk info of client I1121 16:19:58.106960 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:19:58.106968 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e8800 0xc0003e8840] I1121 16:19:58.446393 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:19:59.450495 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:19:59.450569 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:19:59.450594 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:19:59.471117 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:20:06.412890 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:20:06.412991 3798194 memory.go:184] no items to output this cycle I1121 16:20:06.413299 3798194 cpu.go:275] no items to output this cycle E1121 16:20:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:20:16.411986 3798194 memory.go:184] no items to output this cycle I1121 16:20:16.412152 3798194 cpu.go:275] no items to output this cycle E1121 16:20:26.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:20:26.411961 3798194 memory.go:191] Add success. I1121 16:20:26.412344 3798194 cpu.go:282] Add success. I1121 16:20:26.441938 3798194 net.go:642] Add success. I1121 16:20:26.446017 3798194 net.go:778] primary dev: ETH0 I1121 16:20:26.446033 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:20:26.446046 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:20:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:20:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:20:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:20:36.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:20:36.412877 3798194 memory.go:184] no items to output this cycle I1121 16:20:36.413020 3798194 cpu.go:275] no items to output this cycle E1121 16:20:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:20:46.411983 3798194 memory.go:184] no items to output this cycle I1121 16:20:46.412146 3798194 cpu.go:275] no items to output this cycle E1121 16:20:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:20:56.411947 3798194 memory.go:191] Add success. W1121 16:20:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:20:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:20:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:20:56.412327 3798194 cpu.go:282] Add success. I1121 16:20:56.444417 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:20:56.444906 3798194 net.go:642] Add success. I1121 16:20:56.449092 3798194 net.go:778] primary dev: ETH0 I1121 16:20:56.449106 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:20:56.449139 3798194 net.go:659] Add eni metrics success, count: 7 E1121 16:20:57.450159 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 16:20:57.450174 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:20:57.450327 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:20:57.450381 3798194 disk_worker.go:723] disk space is not compliant W1121 16:20:57.450385 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:20:57.458495 3798194 disk_worker.go:509] system disk:vda1 I1121 16:20:57.458537 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:20:58.107076 3798194 disk_info.go:125] begin check local disk info of client I1121 16:20:58.110639 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:20:58.110647 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00035d300 0xc00035d340] E1121 16:20:58.448141 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 16:20:58.448156 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 16:20:59.448605 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 16:20:59.449680 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:20:59.449728 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:20:59.449750 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:20:59.471184 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:21:06.412099 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:21:06.412136 3798194 memory.go:184] no items to output this cycle I1121 16:21:06.412284 3798194 cpu.go:275] no items to output this cycle E1121 16:21:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:21:16.411976 3798194 memory.go:184] no items to output this cycle I1121 16:21:16.412129 3798194 cpu.go:275] no items to output this cycle I1121 16:21:17.508268 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:21:17.508276 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:21:26.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:21:26.412893 3798194 memory.go:191] Add success. I1121 16:21:26.417473 3798194 cpu.go:282] Add success. I1121 16:21:26.442172 3798194 net.go:642] Add success. I1121 16:21:26.443802 3798194 net.go:778] primary dev: ETH0 I1121 16:21:26.443819 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:21:26.443936 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:21:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:21:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:21:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:21:36.412246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:21:36.412270 3798194 memory.go:184] no items to output this cycle I1121 16:21:36.412417 3798194 cpu.go:275] no items to output this cycle E1121 16:21:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:21:46.411983 3798194 memory.go:184] no items to output this cycle I1121 16:21:46.412143 3798194 cpu.go:275] no items to output this cycle E1121 16:21:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:21:56.411944 3798194 memory.go:191] Add success. W1121 16:21:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:21:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:21:56.412000 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:21:56.412340 3798194 cpu.go:282] Add success. I1121 16:21:56.441809 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:21:56.442404 3798194 net.go:642] Add success. I1121 16:21:56.446285 3798194 net.go:778] primary dev: ETH0 I1121 16:21:56.446300 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:21:56.446312 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:21:57.450455 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:21:57.450671 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:21:57.450682 3798194 disk_worker.go:723] disk space is not compliant W1121 16:21:57.450685 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:21:57.458979 3798194 disk_worker.go:509] system disk:vda1 I1121 16:21:57.459025 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:21:58.110746 3798194 disk_info.go:125] begin check local disk info of client I1121 16:21:58.114401 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:21:58.114410 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00035d580 0xc00035d5c0] I1121 16:21:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:21:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:21:59.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:21:59.450119 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:21:59.471640 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:22:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:22:06.412253 3798194 memory.go:184] no items to output this cycle I1121 16:22:06.412404 3798194 cpu.go:275] no items to output this cycle E1121 16:22:16.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:22:16.413255 3798194 memory.go:184] no items to output this cycle I1121 16:22:16.413400 3798194 cpu.go:275] no items to output this cycle E1121 16:22:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:22:26.411948 3798194 memory.go:191] Add success. I1121 16:22:26.412310 3798194 cpu.go:282] Add success. I1121 16:22:26.441883 3798194 net.go:642] Add success. I1121 16:22:26.446230 3798194 net.go:778] primary dev: ETH0 I1121 16:22:26.446245 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:22:26.446258 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:22:29.450039 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:22:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:22:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:22:36.412932 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:22:36.412955 3798194 memory.go:184] no items to output this cycle I1121 16:22:36.413096 3798194 cpu.go:275] no items to output this cycle E1121 16:22:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:22:46.411981 3798194 memory.go:184] no items to output this cycle I1121 16:22:46.412149 3798194 cpu.go:275] no items to output this cycle E1121 16:22:56.411931 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:22:56.412080 3798194 memory.go:191] Add success. W1121 16:22:56.412134 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:22:56.412148 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:22:56.412151 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:22:56.412721 3798194 cpu.go:282] Add success. I1121 16:22:56.440967 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:22:56.441563 3798194 net.go:642] Add success. I1121 16:22:56.446013 3798194 net.go:778] primary dev: ETH0 I1121 16:22:56.446028 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:22:56.446041 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:22:57.450336 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:22:57.450557 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:22:57.450568 3798194 disk_worker.go:723] disk space is not compliant W1121 16:22:57.450571 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:22:57.460802 3798194 disk_worker.go:509] system disk:vda1 I1121 16:22:57.460848 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:22:58.114510 3798194 disk_info.go:125] begin check local disk info of client I1121 16:22:58.117972 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:22:58.117981 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030edc0 0xc00030ee00] I1121 16:22:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:22:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:22:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:22:59.450130 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:22:59.471668 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:23:06.412869 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:23:06.412892 3798194 memory.go:184] no items to output this cycle I1121 16:23:06.413047 3798194 cpu.go:275] no items to output this cycle E1121 16:23:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:23:16.411979 3798194 memory.go:184] no items to output this cycle I1121 16:23:16.412143 3798194 cpu.go:275] no items to output this cycle E1121 16:23:26.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:23:26.412275 3798194 memory.go:191] Add success. I1121 16:23:26.416534 3798194 cpu.go:282] Add success. I1121 16:23:26.442072 3798194 net.go:642] Add success. I1121 16:23:26.443535 3798194 net.go:778] primary dev: ETH0 I1121 16:23:26.443550 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:23:26.443561 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:23:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:23:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:23:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:23:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:23:36.411976 3798194 memory.go:184] no items to output this cycle I1121 16:23:36.412140 3798194 cpu.go:275] no items to output this cycle E1121 16:23:46.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:23:46.412260 3798194 memory.go:184] no items to output this cycle I1121 16:23:46.412402 3798194 cpu.go:275] no items to output this cycle E1121 16:23:56.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:23:56.411959 3798194 memory.go:191] Add success. W1121 16:23:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:23:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:23:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:23:56.412351 3798194 cpu.go:282] Add success. I1121 16:23:56.441803 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:23:56.442312 3798194 net.go:642] Add success. I1121 16:23:56.446366 3798194 net.go:778] primary dev: ETH0 I1121 16:23:56.446381 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:23:56.446394 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:23:57.450343 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:23:57.450548 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:23:57.450559 3798194 disk_worker.go:723] disk space is not compliant W1121 16:23:57.450563 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:23:57.459416 3798194 disk_worker.go:509] system disk:vda1 I1121 16:23:57.459453 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:23:58.118086 3798194 disk_info.go:125] begin check local disk info of client I1121 16:23:58.121672 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:23:58.121680 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e7740 0xc0003e7780] I1121 16:23:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:23:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:23:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:23:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:23:59.471678 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:24:06.411976 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:24:06.411999 3798194 memory.go:184] no items to output this cycle I1121 16:24:06.412158 3798194 cpu.go:275] no items to output this cycle E1121 16:24:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:24:16.411982 3798194 memory.go:184] no items to output this cycle I1121 16:24:16.412136 3798194 cpu.go:275] no items to output this cycle I1121 16:24:17.508438 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:24:17.508446 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:24:26.411555 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:24:26.411589 3798194 memory.go:191] Add success. I1121 16:24:26.411922 3798194 cpu.go:282] Add success. I1121 16:24:26.441571 3798194 net.go:642] Add success. I1121 16:24:26.445470 3798194 net.go:778] primary dev: ETH0 I1121 16:24:26.445484 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:24:26.445498 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:24:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:24:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:24:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:24:36.412815 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:24:36.412837 3798194 memory.go:184] no items to output this cycle I1121 16:24:36.412979 3798194 cpu.go:275] no items to output this cycle E1121 16:24:46.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:24:46.412261 3798194 memory.go:184] no items to output this cycle I1121 16:24:46.412407 3798194 cpu.go:275] no items to output this cycle E1121 16:24:56.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:24:56.412286 3798194 memory.go:191] Add success. W1121 16:24:56.412320 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:24:56.412332 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:24:56.412335 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:24:56.416495 3798194 cpu.go:282] Add success. I1121 16:24:56.441879 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:24:56.442386 3798194 net.go:642] Add success. I1121 16:24:56.444046 3798194 net.go:778] primary dev: ETH0 I1121 16:24:56.444060 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:24:56.444071 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:24:57.450335 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:24:57.450505 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:24:57.450561 3798194 disk_worker.go:723] disk space is not compliant W1121 16:24:57.450564 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:24:57.458557 3798194 disk_worker.go:509] system disk:vda1 I1121 16:24:57.458598 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:24:58.121776 3798194 disk_info.go:125] begin check local disk info of client I1121 16:24:58.125334 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:24:58.125343 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b41c0 0xc0003b4200] I1121 16:24:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:24:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:24:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:24:59.450146 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:24:59.471690 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:25:06.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:25:06.411972 3798194 memory.go:184] no items to output this cycle I1121 16:25:06.412126 3798194 cpu.go:275] no items to output this cycle E1121 16:25:16.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:25:16.413263 3798194 memory.go:184] no items to output this cycle I1121 16:25:16.413405 3798194 cpu.go:275] no items to output this cycle E1121 16:25:26.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:25:26.413261 3798194 memory.go:191] Add success. I1121 16:25:26.418213 3798194 cpu.go:282] Add success. I1121 16:25:26.441908 3798194 net.go:642] Add success. I1121 16:25:26.443384 3798194 net.go:778] primary dev: ETH0 I1121 16:25:26.443400 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:25:26.443414 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:25:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:25:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:25:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:25:36.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:25:36.411993 3798194 memory.go:184] no items to output this cycle I1121 16:25:36.412152 3798194 cpu.go:275] no items to output this cycle E1121 16:25:46.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:25:46.411987 3798194 memory.go:184] no items to output this cycle I1121 16:25:46.412142 3798194 cpu.go:275] no items to output this cycle E1121 16:25:56.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:25:56.413277 3798194 memory.go:191] Add success. W1121 16:25:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:25:56.413323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:25:56.413326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:25:56.417482 3798194 cpu.go:282] Add success. I1121 16:25:56.441645 3798194 event_worker.go:152] Polling the log file for events... I1121 16:25:56.444499 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:25:56.445031 3798194 net.go:642] Add success. I1121 16:25:56.446455 3798194 net.go:778] primary dev: ETH0 I1121 16:25:56.446469 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:25:56.446482 3798194 net.go:659] Add eni metrics success, count: 7 E1121 16:25:57.450489 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 16:25:57.450503 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:25:57.450675 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:25:57.450730 3798194 disk_worker.go:723] disk space is not compliant W1121 16:25:57.450733 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:25:57.458871 3798194 disk_worker.go:509] system disk:vda1 I1121 16:25:57.458912 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:25:58.125445 3798194 disk_info.go:125] begin check local disk info of client I1121 16:25:58.128984 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:25:58.128993 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000402700 0xc000402740] E1121 16:25:58.447750 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 16:25:58.447765 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 16:25:59.448449 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 16:25:59.450579 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:25:59.450631 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:25:59.450650 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:25:59.471128 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:26:06.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:26:06.413260 3798194 memory.go:184] no items to output this cycle I1121 16:26:06.413405 3798194 cpu.go:275] no items to output this cycle E1121 16:26:16.412661 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:26:16.412683 3798194 memory.go:184] no items to output this cycle I1121 16:26:16.412825 3798194 cpu.go:275] no items to output this cycle E1121 16:26:26.411931 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:26:26.411967 3798194 memory.go:191] Add success. I1121 16:26:26.412300 3798194 cpu.go:282] Add success. I1121 16:26:26.441742 3798194 net.go:642] Add success. I1121 16:26:26.445585 3798194 net.go:778] primary dev: ETH0 I1121 16:26:26.445601 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:26:26.445615 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:26:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:26:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:26:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:26:36.413177 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:26:36.413200 3798194 memory.go:184] no items to output this cycle I1121 16:26:36.413355 3798194 cpu.go:275] no items to output this cycle E1121 16:26:46.420234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:26:46.420255 3798194 memory.go:184] no items to output this cycle I1121 16:26:46.420617 3798194 cpu.go:275] no items to output this cycle E1121 16:26:56.411886 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:26:56.411921 3798194 memory.go:191] Add success. W1121 16:26:56.411954 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:26:56.411965 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:26:56.411969 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:26:56.412357 3798194 cpu.go:282] Add success. I1121 16:26:56.441885 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:26:56.442502 3798194 net.go:642] Add success. I1121 16:26:56.446606 3798194 net.go:778] primary dev: ETH0 I1121 16:26:56.446622 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:26:56.446634 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:26:57.448532 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:26:57.448751 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:26:57.448819 3798194 disk_worker.go:723] disk space is not compliant W1121 16:26:57.448822 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:26:57.451358 3798194 disk_worker.go:509] system disk:vda1 I1121 16:26:57.451398 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:26:58.131144 3798194 disk_info.go:125] begin check local disk info of client I1121 16:26:58.134716 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:26:58.134725 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000314340 0xc000314380] I1121 16:26:58.445986 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:26:59.450029 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:26:59.450142 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:26:59.450172 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:26:59.470684 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:27:06.413261 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:27:06.413290 3798194 memory.go:184] no items to output this cycle I1121 16:27:06.413441 3798194 cpu.go:275] no items to output this cycle E1121 16:27:16.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:27:16.411989 3798194 memory.go:184] no items to output this cycle I1121 16:27:16.412149 3798194 cpu.go:275] no items to output this cycle I1121 16:27:17.509394 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:27:17.509402 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:27:26.411304 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:27:26.411337 3798194 memory.go:191] Add success. I1121 16:27:26.411693 3798194 cpu.go:282] Add success. I1121 16:27:26.441307 3798194 net.go:642] Add success. I1121 16:27:26.445170 3798194 net.go:778] primary dev: ETH0 I1121 16:27:26.445186 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:27:26.445199 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:27:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:27:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:27:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:27:36.412359 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:27:36.412383 3798194 memory.go:184] no items to output this cycle I1121 16:27:36.412526 3798194 cpu.go:275] no items to output this cycle E1121 16:27:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:27:46.411977 3798194 memory.go:184] no items to output this cycle I1121 16:27:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 16:27:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:27:56.412267 3798194 memory.go:191] Add success. W1121 16:27:56.412295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:27:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:27:56.412310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:27:56.416301 3798194 cpu.go:282] Add success. I1121 16:27:56.441690 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:27:56.442195 3798194 net.go:642] Add success. I1121 16:27:56.443585 3798194 net.go:778] primary dev: ETH0 I1121 16:27:56.443600 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:27:56.443613 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:27:57.450632 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:27:57.451018 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:27:57.451031 3798194 disk_worker.go:723] disk space is not compliant W1121 16:27:57.451034 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:27:57.459860 3798194 disk_worker.go:509] system disk:vda1 I1121 16:27:57.459902 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:27:58.134831 3798194 disk_info.go:125] begin check local disk info of client I1121 16:27:58.138397 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:27:58.138406 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000491400 0xc000491440] I1121 16:27:58.446589 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:27:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:27:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:27:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:27:59.471658 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:28:06.411996 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:28:06.412019 3798194 memory.go:184] no items to output this cycle I1121 16:28:06.412179 3798194 cpu.go:275] no items to output this cycle E1121 16:28:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:28:16.411985 3798194 memory.go:184] no items to output this cycle I1121 16:28:16.412142 3798194 cpu.go:275] no items to output this cycle E1121 16:28:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:28:26.411958 3798194 memory.go:191] Add success. I1121 16:28:26.412312 3798194 cpu.go:282] Add success. I1121 16:28:26.441927 3798194 net.go:642] Add success. I1121 16:28:26.445739 3798194 net.go:778] primary dev: ETH0 I1121 16:28:26.445756 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:28:26.445771 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:28:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:28:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:28:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:28:36.412758 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:28:36.412782 3798194 memory.go:184] no items to output this cycle I1121 16:28:36.412941 3798194 cpu.go:275] no items to output this cycle E1121 16:28:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:28:46.412254 3798194 memory.go:184] no items to output this cycle I1121 16:28:46.412412 3798194 cpu.go:275] no items to output this cycle E1121 16:28:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:28:56.411952 3798194 memory.go:191] Add success. W1121 16:28:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:28:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:28:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:28:56.412323 3798194 cpu.go:282] Add success. I1121 16:28:56.441715 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:28:56.442320 3798194 net.go:642] Add success. I1121 16:28:56.446425 3798194 net.go:778] primary dev: ETH0 I1121 16:28:56.446440 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:28:56.446453 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:28:57.450419 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:28:57.450628 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:28:57.450640 3798194 disk_worker.go:723] disk space is not compliant W1121 16:28:57.450644 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:28:57.458837 3798194 disk_worker.go:509] system disk:vda1 I1121 16:28:57.458882 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:28:58.138510 3798194 disk_info.go:125] begin check local disk info of client I1121 16:28:58.142041 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:28:58.142050 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a8b00 0xc0003a8b40] I1121 16:28:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:28:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:28:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:28:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:28:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:29:06.413199 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:29:06.413222 3798194 memory.go:184] no items to output this cycle I1121 16:29:06.413363 3798194 cpu.go:275] no items to output this cycle E1121 16:29:16.411978 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:29:16.412000 3798194 memory.go:184] no items to output this cycle I1121 16:29:16.412158 3798194 cpu.go:275] no items to output this cycle E1121 16:29:26.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:29:26.411974 3798194 memory.go:191] Add success. I1121 16:29:26.412353 3798194 cpu.go:282] Add success. I1121 16:29:26.442023 3798194 net.go:642] Add success. I1121 16:29:26.446242 3798194 net.go:778] primary dev: ETH0 I1121 16:29:26.446257 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:29:26.446270 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:29:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:29:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:29:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:29:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:29:36.413254 3798194 memory.go:184] no items to output this cycle I1121 16:29:36.413410 3798194 cpu.go:275] no items to output this cycle E1121 16:29:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:29:46.413258 3798194 memory.go:184] no items to output this cycle I1121 16:29:46.413407 3798194 cpu.go:275] no items to output this cycle E1121 16:29:56.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:29:56.412270 3798194 memory.go:191] Add success. W1121 16:29:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:29:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:29:56.412314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:29:56.416294 3798194 cpu.go:282] Add success. I1121 16:29:56.441749 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:29:56.442258 3798194 net.go:642] Add success. I1121 16:29:56.443680 3798194 net.go:778] primary dev: ETH0 I1121 16:29:56.443695 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:29:56.443707 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:29:57.450517 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:29:57.450689 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:29:57.450742 3798194 disk_worker.go:723] disk space is not compliant W1121 16:29:57.450745 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:29:57.458860 3798194 disk_worker.go:509] system disk:vda1 I1121 16:29:57.458904 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:29:58.142174 3798194 disk_info.go:125] begin check local disk info of client I1121 16:29:58.145670 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:29:58.145679 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030d300 0xc00030d340] I1121 16:29:58.446016 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:29:59.450618 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:29:59.450703 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:29:59.450728 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:29:59.471236 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:30:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:30:06.411984 3798194 memory.go:184] no items to output this cycle I1121 16:30:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 16:30:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:30:16.411973 3798194 memory.go:184] no items to output this cycle I1121 16:30:16.412132 3798194 cpu.go:275] no items to output this cycle I1121 16:30:17.510396 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:30:17.510409 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:30:26.411467 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:30:26.411501 3798194 memory.go:191] Add success. I1121 16:30:26.411842 3798194 cpu.go:282] Add success. I1121 16:30:26.441520 3798194 net.go:642] Add success. I1121 16:30:26.445696 3798194 net.go:778] primary dev: ETH0 I1121 16:30:26.445711 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:30:26.445724 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:30:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:30:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:30:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:30:36.413171 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:30:36.413193 3798194 memory.go:184] no items to output this cycle I1121 16:30:36.413343 3798194 cpu.go:275] no items to output this cycle E1121 16:30:46.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:30:46.413267 3798194 memory.go:184] no items to output this cycle I1121 16:30:46.413414 3798194 cpu.go:275] no items to output this cycle E1121 16:30:56.412036 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:30:56.412164 3798194 memory.go:191] Add success. W1121 16:30:56.412202 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:30:56.412214 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:30:56.412218 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:30:56.412889 3798194 cpu.go:282] Add success. I1121 16:30:56.445242 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:30:56.445825 3798194 net.go:642] Add success. I1121 16:30:56.449579 3798194 net.go:778] primary dev: ETH0 I1121 16:30:56.449594 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:30:56.449607 3798194 net.go:659] Add eni metrics success, count: 7 E1121 16:30:57.451957 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 16:30:57.451971 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:30:57.452163 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:30:57.452231 3798194 disk_worker.go:723] disk space is not compliant W1121 16:30:57.452235 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:30:57.465251 3798194 disk_worker.go:509] system disk:vda1 I1121 16:30:57.465295 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:30:58.148538 3798194 disk_info.go:125] begin check local disk info of client I1121 16:30:58.152210 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:30:58.152218 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ec00 0xc00030ec40] E1121 16:30:58.448214 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 16:30:58.448229 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 16:30:59.449442 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 16:30:59.450517 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:30:59.450576 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:30:59.450597 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:30:59.471105 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:31:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:31:06.411983 3798194 memory.go:184] no items to output this cycle I1121 16:31:06.412157 3798194 cpu.go:275] no items to output this cycle E1121 16:31:16.412343 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:31:16.412365 3798194 memory.go:184] no items to output this cycle I1121 16:31:16.412526 3798194 cpu.go:275] no items to output this cycle E1121 16:31:26.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:31:26.413274 3798194 memory.go:191] Add success. I1121 16:31:26.417715 3798194 cpu.go:282] Add success. I1121 16:31:26.441684 3798194 net.go:642] Add success. I1121 16:31:26.443167 3798194 net.go:778] primary dev: ETH0 I1121 16:31:26.443183 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:31:26.443194 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:31:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:31:29.450135 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:31:29.450167 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:31:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:31:36.411979 3798194 memory.go:184] no items to output this cycle I1121 16:31:36.412137 3798194 cpu.go:275] no items to output this cycle E1121 16:31:46.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:31:46.413251 3798194 memory.go:184] no items to output this cycle I1121 16:31:46.413409 3798194 cpu.go:275] no items to output this cycle E1121 16:31:56.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:31:56.413279 3798194 memory.go:191] Add success. W1121 16:31:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:31:56.413324 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:31:56.413327 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:31:56.417673 3798194 cpu.go:282] Add success. I1121 16:31:56.440978 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:31:56.441568 3798194 net.go:642] Add success. I1121 16:31:56.443170 3798194 net.go:778] primary dev: ETH0 I1121 16:31:56.443184 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:31:56.443197 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:31:57.450342 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:31:57.450559 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:31:57.450569 3798194 disk_worker.go:723] disk space is not compliant W1121 16:31:57.450573 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:31:57.459511 3798194 disk_worker.go:509] system disk:vda1 I1121 16:31:57.459554 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:31:58.154153 3798194 disk_info.go:125] begin check local disk info of client I1121 16:31:58.157718 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:31:58.157726 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00035e500 0xc00035e540] I1121 16:31:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:31:59.450098 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:31:59.450194 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:31:59.450220 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:31:59.470702 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:32:06.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:32:06.411972 3798194 memory.go:184] no items to output this cycle I1121 16:32:06.412137 3798194 cpu.go:275] no items to output this cycle E1121 16:32:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:32:16.413259 3798194 memory.go:184] no items to output this cycle I1121 16:32:16.413406 3798194 cpu.go:275] no items to output this cycle E1121 16:32:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:32:26.413267 3798194 memory.go:191] Add success. I1121 16:32:26.417300 3798194 cpu.go:282] Add success. I1121 16:32:26.441732 3798194 net.go:642] Add success. I1121 16:32:26.443247 3798194 net.go:778] primary dev: ETH0 I1121 16:32:26.443262 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:32:26.443274 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:32:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:32:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:32:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:32:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:32:36.412260 3798194 memory.go:184] no items to output this cycle I1121 16:32:36.412409 3798194 cpu.go:275] no items to output this cycle E1121 16:32:46.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:32:46.412871 3798194 memory.go:184] no items to output this cycle I1121 16:32:46.413021 3798194 cpu.go:275] no items to output this cycle E1121 16:32:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:32:56.412269 3798194 memory.go:191] Add success. W1121 16:32:56.412298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:32:56.412310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:32:56.412313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:32:56.416474 3798194 cpu.go:282] Add success. I1121 16:32:56.441862 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:32:56.442383 3798194 net.go:642] Add success. I1121 16:32:56.443879 3798194 net.go:778] primary dev: ETH0 I1121 16:32:56.443897 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:32:56.443910 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:32:57.450884 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:32:57.451089 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:32:57.451102 3798194 disk_worker.go:723] disk space is not compliant W1121 16:32:57.451105 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:32:57.459103 3798194 disk_worker.go:509] system disk:vda1 I1121 16:32:57.459159 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:32:58.157828 3798194 disk_info.go:125] begin check local disk info of client I1121 16:32:58.161942 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:32:58.161951 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d2040 0xc0003d2080] I1121 16:32:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:32:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:32:59.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:32:59.450495 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:32:59.471011 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:33:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:33:06.411976 3798194 memory.go:184] no items to output this cycle I1121 16:33:06.412132 3798194 cpu.go:275] no items to output this cycle E1121 16:33:16.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:33:16.411991 3798194 memory.go:184] no items to output this cycle I1121 16:33:16.412174 3798194 cpu.go:275] no items to output this cycle I1121 16:33:17.511403 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:33:17.511411 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:33:26.411432 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:33:26.411467 3798194 memory.go:191] Add success. I1121 16:33:26.411827 3798194 cpu.go:282] Add success. I1121 16:33:26.441862 3798194 net.go:642] Add success. I1121 16:33:26.446354 3798194 net.go:778] primary dev: ETH0 I1121 16:33:26.446371 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:33:26.446395 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:33:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:33:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:33:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:33:36.412510 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:33:36.412536 3798194 memory.go:184] no items to output this cycle I1121 16:33:36.412681 3798194 cpu.go:275] no items to output this cycle E1121 16:33:46.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:33:46.411992 3798194 memory.go:184] no items to output this cycle I1121 16:33:46.412167 3798194 cpu.go:275] no items to output this cycle E1121 16:33:56.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:33:56.413280 3798194 memory.go:191] Add success. W1121 16:33:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:33:56.413323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:33:56.413326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:33:56.417798 3798194 cpu.go:282] Add success. I1121 16:33:56.441207 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:33:56.441787 3798194 net.go:642] Add success. I1121 16:33:56.443311 3798194 net.go:778] primary dev: ETH0 I1121 16:33:56.443326 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:33:56.443338 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:33:57.451025 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:33:57.451256 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:33:57.451267 3798194 disk_worker.go:723] disk space is not compliant W1121 16:33:57.451271 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:33:57.459308 3798194 disk_worker.go:509] system disk:vda1 I1121 16:33:57.459352 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:33:58.162055 3798194 disk_info.go:125] begin check local disk info of client I1121 16:33:58.165670 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:33:58.165679 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000464480 0xc0004644c0] I1121 16:33:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:33:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:33:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:33:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:33:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:34:06.412868 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:34:06.412891 3798194 memory.go:184] no items to output this cycle I1121 16:34:06.413047 3798194 cpu.go:275] no items to output this cycle E1121 16:34:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:34:16.411987 3798194 memory.go:184] no items to output this cycle I1121 16:34:16.412142 3798194 cpu.go:275] no items to output this cycle E1121 16:34:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:34:26.411951 3798194 memory.go:191] Add success. I1121 16:34:26.412306 3798194 cpu.go:282] Add success. I1121 16:34:26.441735 3798194 net.go:642] Add success. I1121 16:34:26.445677 3798194 net.go:778] primary dev: ETH0 I1121 16:34:26.445692 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:34:26.445704 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:34:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:34:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:34:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:34:36.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:34:36.412261 3798194 memory.go:184] no items to output this cycle I1121 16:34:36.412403 3798194 cpu.go:275] no items to output this cycle E1121 16:34:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:34:46.411976 3798194 memory.go:184] no items to output this cycle I1121 16:34:46.412138 3798194 cpu.go:275] no items to output this cycle E1121 16:34:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:34:56.412381 3798194 memory.go:191] Add success. W1121 16:34:56.412413 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:34:56.412426 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:34:56.412429 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:34:56.417492 3798194 cpu.go:282] Add success. I1121 16:34:56.441136 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:34:56.441614 3798194 net.go:642] Add success. I1121 16:34:56.443087 3798194 net.go:778] primary dev: ETH0 I1121 16:34:56.443102 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:34:56.443132 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:34:57.450402 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:34:57.450561 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:34:57.450612 3798194 disk_worker.go:723] disk space is not compliant W1121 16:34:57.450615 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:34:57.458814 3798194 disk_worker.go:509] system disk:vda1 I1121 16:34:57.458858 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:34:58.165784 3798194 disk_info.go:125] begin check local disk info of client I1121 16:34:58.169378 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:34:58.169387 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ee00 0xc00030ee40] I1121 16:34:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:34:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:34:59.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:34:59.450148 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:34:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:35:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:35:06.411981 3798194 memory.go:184] no items to output this cycle I1121 16:35:06.412142 3798194 cpu.go:275] no items to output this cycle E1121 16:35:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:35:16.411973 3798194 memory.go:184] no items to output this cycle I1121 16:35:16.412139 3798194 cpu.go:275] no items to output this cycle E1121 16:35:26.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:35:26.413270 3798194 memory.go:191] Add success. I1121 16:35:26.417530 3798194 cpu.go:282] Add success. I1121 16:35:26.442069 3798194 net.go:642] Add success. I1121 16:35:26.443426 3798194 net.go:778] primary dev: ETH0 I1121 16:35:26.443440 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:35:26.443453 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:35:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:35:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:35:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:35:36.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:35:36.412259 3798194 memory.go:184] no items to output this cycle I1121 16:35:36.412405 3798194 cpu.go:275] no items to output this cycle E1121 16:35:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:35:46.413261 3798194 memory.go:184] no items to output this cycle I1121 16:35:46.413405 3798194 cpu.go:275] no items to output this cycle E1121 16:35:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:35:56.411949 3798194 memory.go:191] Add success. W1121 16:35:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:35:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:35:56.411993 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:35:56.412339 3798194 cpu.go:282] Add success. I1121 16:35:56.441586 3798194 event_worker.go:152] Polling the log file for events... I1121 16:35:56.444536 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:35:56.445005 3798194 net.go:642] Add success. I1121 16:35:56.449003 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795248356, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 16:35:56.449131 3798194 net.go:778] primary dev: ETH0 I1121 16:35:56.449145 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:35:56.449159 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:35:56.460524 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 16:35:57.445221 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 16:35:57.445239 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:35:57.445449 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:35:57.445512 3798194 disk_worker.go:723] disk space is not compliant W1121 16:35:57.445515 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:35:57.447948 3798194 disk_worker.go:509] system disk:vda1 I1121 16:35:57.447986 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:35:58.169494 3798194 disk_info.go:125] begin check local disk info of client I1121 16:35:58.173044 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:35:58.173053 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000416540 0xc000416580] E1121 16:35:58.447808 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 16:35:58.447822 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 16:35:59.448617 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 16:35:59.449659 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:35:59.449712 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:35:59.449736 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:35:59.471203 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:36:06.412828 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:36:06.412858 3798194 memory.go:184] no items to output this cycle I1121 16:36:06.413005 3798194 cpu.go:275] no items to output this cycle E1121 16:36:16.411944 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:36:16.411965 3798194 memory.go:184] no items to output this cycle I1121 16:36:16.412123 3798194 cpu.go:275] no items to output this cycle I1121 16:36:17.512406 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:36:17.512414 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:36:26.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:36:26.413257 3798194 memory.go:191] Add success. I1121 16:36:26.417787 3798194 cpu.go:282] Add success. I1121 16:36:26.441286 3798194 net.go:642] Add success. I1121 16:36:26.442797 3798194 net.go:778] primary dev: ETH0 I1121 16:36:26.442813 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:36:26.442826 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:36:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:36:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:36:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:36:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:36:36.413254 3798194 memory.go:184] no items to output this cycle I1121 16:36:36.413402 3798194 cpu.go:275] no items to output this cycle E1121 16:36:46.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:36:46.413259 3798194 memory.go:184] no items to output this cycle I1121 16:36:46.413408 3798194 cpu.go:275] no items to output this cycle E1121 16:36:56.414302 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:36:56.414349 3798194 memory.go:191] Add success. W1121 16:36:56.414394 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:36:56.414413 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:36:56.414418 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:36:56.419057 3798194 cpu.go:282] Add success. I1121 16:36:56.440937 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:36:56.441567 3798194 net.go:642] Add success. I1121 16:36:56.443022 3798194 net.go:778] primary dev: ETH0 I1121 16:36:56.443036 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:36:56.443049 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:36:57.445257 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:36:57.445539 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:36:57.445552 3798194 disk_worker.go:723] disk space is not compliant W1121 16:36:57.445555 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:36:57.448036 3798194 disk_worker.go:509] system disk:vda1 I1121 16:36:57.448074 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:36:58.173148 3798194 disk_info.go:125] begin check local disk info of client I1121 16:36:58.176736 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:36:58.176746 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037fa40 0xc00037fa80] I1121 16:36:58.445989 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:36:59.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:36:59.450122 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:36:59.450149 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:36:59.471650 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:37:06.412774 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:37:06.412797 3798194 memory.go:184] no items to output this cycle I1121 16:37:06.412938 3798194 cpu.go:275] no items to output this cycle E1121 16:37:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:37:16.411981 3798194 memory.go:184] no items to output this cycle I1121 16:37:16.412135 3798194 cpu.go:275] no items to output this cycle E1121 16:37:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:37:26.411960 3798194 memory.go:191] Add success. I1121 16:37:26.412509 3798194 cpu.go:282] Add success. I1121 16:37:26.441315 3798194 net.go:642] Add success. I1121 16:37:26.445419 3798194 net.go:778] primary dev: ETH0 I1121 16:37:26.445434 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:37:26.445446 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:37:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:37:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:37:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:37:36.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:37:36.411970 3798194 memory.go:184] no items to output this cycle I1121 16:37:36.412122 3798194 cpu.go:275] no items to output this cycle E1121 16:37:46.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:37:46.412880 3798194 memory.go:184] no items to output this cycle I1121 16:37:46.413030 3798194 cpu.go:275] no items to output this cycle E1121 16:37:56.411934 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:37:56.411967 3798194 memory.go:191] Add success. W1121 16:37:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:37:56.412011 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:37:56.412014 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:37:56.412386 3798194 cpu.go:282] Add success. I1121 16:37:56.441861 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:37:56.442411 3798194 net.go:642] Add success. I1121 16:37:56.446512 3798194 net.go:778] primary dev: ETH0 I1121 16:37:56.446527 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:37:56.446540 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:37:57.450420 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:37:57.450634 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:37:57.450645 3798194 disk_worker.go:723] disk space is not compliant W1121 16:37:57.450648 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:37:57.458804 3798194 disk_worker.go:509] system disk:vda1 I1121 16:37:57.458846 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:37:58.176855 3798194 disk_info.go:125] begin check local disk info of client I1121 16:37:58.180386 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:37:58.180395 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d7f00 0xc0003d7f40] I1121 16:37:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:37:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:37:59.450122 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:37:59.450158 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:37:59.471708 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:38:06.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:38:06.412257 3798194 memory.go:184] no items to output this cycle I1121 16:38:06.412397 3798194 cpu.go:275] no items to output this cycle E1121 16:38:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:38:16.413259 3798194 memory.go:184] no items to output this cycle I1121 16:38:16.413404 3798194 cpu.go:275] no items to output this cycle E1121 16:38:26.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:38:26.412269 3798194 memory.go:191] Add success. I1121 16:38:26.416400 3798194 cpu.go:282] Add success. I1121 16:38:26.441949 3798194 net.go:642] Add success. I1121 16:38:26.443421 3798194 net.go:778] primary dev: ETH0 I1121 16:38:26.443436 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:38:26.443452 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:38:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:38:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:38:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:38:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:38:36.412255 3798194 memory.go:184] no items to output this cycle I1121 16:38:36.412396 3798194 cpu.go:275] no items to output this cycle E1121 16:38:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:38:46.411976 3798194 memory.go:184] no items to output this cycle I1121 16:38:46.412137 3798194 cpu.go:275] no items to output this cycle E1121 16:38:56.411938 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:38:56.411977 3798194 memory.go:191] Add success. W1121 16:38:56.412008 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:38:56.412020 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:38:56.412023 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:38:56.412381 3798194 cpu.go:282] Add success. I1121 16:38:56.441850 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:38:56.442418 3798194 net.go:642] Add success. I1121 16:38:56.446569 3798194 net.go:778] primary dev: ETH0 I1121 16:38:56.446584 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:38:56.446597 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:38:57.450304 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:38:57.450476 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:38:57.450533 3798194 disk_worker.go:723] disk space is not compliant W1121 16:38:57.450536 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:38:57.458622 3798194 disk_worker.go:509] system disk:vda1 I1121 16:38:57.458676 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:38:58.180498 3798194 disk_info.go:125] begin check local disk info of client I1121 16:38:58.184209 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:38:58.184219 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0140 0xc0002e0180] I1121 16:38:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:38:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:38:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:38:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:38:59.471645 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:39:06.412768 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:39:06.412791 3798194 memory.go:184] no items to output this cycle I1121 16:39:06.412944 3798194 cpu.go:275] no items to output this cycle E1121 16:39:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:39:16.411985 3798194 memory.go:184] no items to output this cycle I1121 16:39:16.412142 3798194 cpu.go:275] no items to output this cycle I1121 16:39:17.513411 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:39:17.513419 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:39:26.411280 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:39:26.411311 3798194 memory.go:191] Add success. I1121 16:39:26.411645 3798194 cpu.go:282] Add success. I1121 16:39:26.442286 3798194 net.go:642] Add success. I1121 16:39:26.446216 3798194 net.go:778] primary dev: ETH0 I1121 16:39:26.446232 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:39:26.446244 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:39:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:39:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:39:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:39:36.412302 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:39:36.412331 3798194 memory.go:184] no items to output this cycle I1121 16:39:36.412476 3798194 cpu.go:275] no items to output this cycle E1121 16:39:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:39:46.411982 3798194 memory.go:184] no items to output this cycle I1121 16:39:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 16:39:56.412841 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:39:56.412875 3798194 memory.go:191] Add success. W1121 16:39:56.412906 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:39:56.412919 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:39:56.412922 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:39:56.417946 3798194 cpu.go:282] Add success. I1121 16:39:56.441428 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:39:56.441899 3798194 net.go:642] Add success. I1121 16:39:56.443391 3798194 net.go:778] primary dev: ETH0 I1121 16:39:56.443405 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:39:56.443418 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:39:57.447901 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:39:57.448127 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:39:57.448189 3798194 disk_worker.go:723] disk space is not compliant W1121 16:39:57.448192 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:39:57.450927 3798194 disk_worker.go:509] system disk:vda1 I1121 16:39:57.450968 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:39:58.184326 3798194 disk_info.go:125] begin check local disk info of client I1121 16:39:58.187930 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:39:58.187939 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004682c0 0xc000468300] I1121 16:39:58.445984 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:39:59.450495 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:39:59.450577 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:39:59.450602 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:39:59.471130 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:40:06.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:40:06.412887 3798194 memory.go:184] no items to output this cycle I1121 16:40:06.413032 3798194 cpu.go:275] no items to output this cycle E1121 16:40:16.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:40:16.411985 3798194 memory.go:184] no items to output this cycle I1121 16:40:16.412143 3798194 cpu.go:275] no items to output this cycle E1121 16:40:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:40:26.411949 3798194 memory.go:191] Add success. I1121 16:40:26.412298 3798194 cpu.go:282] Add success. I1121 16:40:26.441895 3798194 net.go:642] Add success. I1121 16:40:26.446073 3798194 net.go:778] primary dev: ETH0 I1121 16:40:26.446087 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:40:26.446100 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:40:29.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:40:29.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:40:29.450492 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:40:36.413211 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:40:36.413238 3798194 memory.go:184] no items to output this cycle I1121 16:40:36.413382 3798194 cpu.go:275] no items to output this cycle E1121 16:40:46.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:40:46.412259 3798194 memory.go:184] no items to output this cycle I1121 16:40:46.412407 3798194 cpu.go:275] no items to output this cycle E1121 16:40:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:40:56.411951 3798194 memory.go:191] Add success. W1121 16:40:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:40:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:40:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:40:56.412366 3798194 cpu.go:282] Add success. I1121 16:40:56.444398 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:40:56.444982 3798194 net.go:642] Add success. I1121 16:40:56.449798 3798194 net.go:778] primary dev: ETH0 I1121 16:40:56.449813 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:40:56.449827 3798194 net.go:659] Add eni metrics success, count: 7 E1121 16:40:57.450800 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 16:40:57.450814 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:40:57.451001 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:40:57.451010 3798194 disk_worker.go:723] disk space is not compliant W1121 16:40:57.451013 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:40:57.459236 3798194 disk_worker.go:509] system disk:vda1 I1121 16:40:57.459279 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:40:58.188038 3798194 disk_info.go:125] begin check local disk info of client I1121 16:40:58.191574 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:40:58.191583 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ad6c0 0xc0003ad700] E1121 16:40:58.447781 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 16:40:58.447796 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 16:40:59.448619 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 16:40:59.449694 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:40:59.449744 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:40:59.449762 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:40:59.471185 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:41:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:41:06.412260 3798194 memory.go:184] no items to output this cycle I1121 16:41:06.412407 3798194 cpu.go:275] no items to output this cycle E1121 16:41:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:41:16.411977 3798194 memory.go:184] no items to output this cycle I1121 16:41:16.412144 3798194 cpu.go:275] no items to output this cycle E1121 16:41:26.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:41:26.411945 3798194 memory.go:191] Add success. I1121 16:41:26.412297 3798194 cpu.go:282] Add success. I1121 16:41:26.442209 3798194 net.go:642] Add success. I1121 16:41:26.448137 3798194 net.go:778] primary dev: ETH0 I1121 16:41:26.448155 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:41:26.448273 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:41:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:41:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:41:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:41:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:41:36.413264 3798194 memory.go:184] no items to output this cycle I1121 16:41:36.413407 3798194 cpu.go:275] no items to output this cycle E1121 16:41:46.412838 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:41:46.412861 3798194 memory.go:184] no items to output this cycle I1121 16:41:46.413008 3798194 cpu.go:275] no items to output this cycle E1121 16:41:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:41:56.411953 3798194 memory.go:191] Add success. W1121 16:41:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:41:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:41:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:41:56.412336 3798194 cpu.go:282] Add success. I1121 16:41:56.441796 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:41:56.442371 3798194 net.go:642] Add success. I1121 16:41:56.446235 3798194 net.go:778] primary dev: ETH0 I1121 16:41:56.446249 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:41:56.446263 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:41:57.450271 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:41:57.450475 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:41:57.450486 3798194 disk_worker.go:723] disk space is not compliant W1121 16:41:57.450489 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:41:57.458712 3798194 disk_worker.go:509] system disk:vda1 I1121 16:41:57.458757 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:41:58.191688 3798194 disk_info.go:125] begin check local disk info of client I1121 16:41:58.195383 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:41:58.195392 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040bbc0 0xc00040bc00] I1121 16:41:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:41:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:41:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:41:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:41:59.471651 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:42:06.413658 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:42:06.413687 3798194 memory.go:184] no items to output this cycle I1121 16:42:06.413830 3798194 cpu.go:275] no items to output this cycle E1121 16:42:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:42:16.411974 3798194 memory.go:184] no items to output this cycle I1121 16:42:16.412131 3798194 cpu.go:275] no items to output this cycle I1121 16:42:17.514419 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:42:17.514428 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:42:26.411191 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:42:26.411235 3798194 memory.go:191] Add success. I1121 16:42:26.411574 3798194 cpu.go:282] Add success. I1121 16:42:26.442265 3798194 net.go:642] Add success. I1121 16:42:26.446030 3798194 net.go:778] primary dev: ETH0 I1121 16:42:26.446046 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:42:26.446059 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:42:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:42:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:42:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:42:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:42:36.413255 3798194 memory.go:184] no items to output this cycle I1121 16:42:36.413397 3798194 cpu.go:275] no items to output this cycle E1121 16:42:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:42:46.412256 3798194 memory.go:184] no items to output this cycle I1121 16:42:46.412402 3798194 cpu.go:275] no items to output this cycle E1121 16:42:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:42:56.412264 3798194 memory.go:191] Add success. W1121 16:42:56.412291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:42:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:42:56.412306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:42:56.416852 3798194 cpu.go:282] Add success. I1121 16:42:56.441368 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:42:56.442349 3798194 net.go:642] Add success. I1121 16:42:56.444134 3798194 net.go:778] primary dev: ETH0 I1121 16:42:56.444154 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:42:56.444168 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:42:57.450350 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:42:57.450563 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:42:57.450576 3798194 disk_worker.go:723] disk space is not compliant W1121 16:42:57.450579 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:42:57.458579 3798194 disk_worker.go:509] system disk:vda1 I1121 16:42:57.458620 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:42:58.195500 3798194 disk_info.go:125] begin check local disk info of client I1121 16:42:58.199032 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:42:58.199042 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004c3880 0xc0004c38c0] I1121 16:42:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:42:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:42:59.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:42:59.450133 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:42:59.471659 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:43:06.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:43:06.411983 3798194 memory.go:184] no items to output this cycle I1121 16:43:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 16:43:16.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:43:16.411970 3798194 memory.go:184] no items to output this cycle I1121 16:43:16.412127 3798194 cpu.go:275] no items to output this cycle E1121 16:43:26.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:43:26.411949 3798194 memory.go:191] Add success. I1121 16:43:26.412296 3798194 cpu.go:282] Add success. I1121 16:43:26.441925 3798194 net.go:642] Add success. I1121 16:43:26.446534 3798194 net.go:778] primary dev: ETH0 I1121 16:43:26.446548 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:43:26.446562 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:43:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:43:29.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:43:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:43:36.412761 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:43:36.412784 3798194 memory.go:184] no items to output this cycle I1121 16:43:36.412932 3798194 cpu.go:275] no items to output this cycle E1121 16:43:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:43:46.411981 3798194 memory.go:184] no items to output this cycle I1121 16:43:46.412138 3798194 cpu.go:275] no items to output this cycle E1121 16:43:56.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:43:56.412279 3798194 memory.go:191] Add success. W1121 16:43:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:43:56.412318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:43:56.412321 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:43:56.416830 3798194 cpu.go:282] Add success. I1121 16:43:56.441425 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:43:56.442048 3798194 net.go:642] Add success. I1121 16:43:56.443618 3798194 net.go:778] primary dev: ETH0 I1121 16:43:56.443632 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:43:56.443644 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:43:57.450271 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:43:57.450468 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:43:57.450478 3798194 disk_worker.go:723] disk space is not compliant W1121 16:43:57.450482 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:43:57.458555 3798194 disk_worker.go:509] system disk:vda1 I1121 16:43:57.458601 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:43:58.199143 3798194 disk_info.go:125] begin check local disk info of client I1121 16:43:58.202668 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:43:58.202677 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004027c0 0xc000402800] I1121 16:43:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:43:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:43:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:43:59.450126 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:43:59.471636 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:44:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:44:06.411982 3798194 memory.go:184] no items to output this cycle I1121 16:44:06.412136 3798194 cpu.go:275] no items to output this cycle E1121 16:44:16.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:44:16.412264 3798194 memory.go:184] no items to output this cycle I1121 16:44:16.412409 3798194 cpu.go:275] no items to output this cycle E1121 16:44:26.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:44:26.412275 3798194 memory.go:191] Add success. I1121 16:44:26.417001 3798194 cpu.go:282] Add success. I1121 16:44:26.441513 3798194 net.go:642] Add success. I1121 16:44:26.442949 3798194 net.go:778] primary dev: ETH0 I1121 16:44:26.442963 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:44:26.442975 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:44:29.450411 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:44:29.450487 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:44:29.450515 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:44:36.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:44:36.412267 3798194 memory.go:184] no items to output this cycle I1121 16:44:36.412410 3798194 cpu.go:275] no items to output this cycle E1121 16:44:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:44:46.411983 3798194 memory.go:184] no items to output this cycle I1121 16:44:46.412144 3798194 cpu.go:275] no items to output this cycle E1121 16:44:56.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:44:56.412290 3798194 memory.go:191] Add success. W1121 16:44:56.412323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:44:56.412334 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:44:56.412338 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:44:56.417557 3798194 cpu.go:282] Add success. I1121 16:44:56.441953 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:44:56.442546 3798194 net.go:642] Add success. I1121 16:44:56.444121 3798194 net.go:778] primary dev: ETH0 I1121 16:44:56.444136 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:44:56.444148 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:44:57.450337 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:44:57.450566 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:44:57.450578 3798194 disk_worker.go:723] disk space is not compliant W1121 16:44:57.450581 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:44:57.458798 3798194 disk_worker.go:509] system disk:vda1 I1121 16:44:57.458843 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:44:58.205145 3798194 disk_info.go:125] begin check local disk info of client I1121 16:44:58.208679 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:44:58.208688 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00038a200 0xc00038a240] I1121 16:44:58.445997 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:44:59.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:44:59.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:44:59.450508 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:44:59.471008 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:45:06.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:45:06.411982 3798194 memory.go:184] no items to output this cycle I1121 16:45:06.412139 3798194 cpu.go:275] no items to output this cycle E1121 16:45:16.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:45:16.412879 3798194 memory.go:184] no items to output this cycle I1121 16:45:16.413028 3798194 cpu.go:275] no items to output this cycle I1121 16:45:17.515178 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:45:17.515186 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:45:26.411388 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:45:26.411420 3798194 memory.go:191] Add success. I1121 16:45:26.411743 3798194 cpu.go:282] Add success. I1121 16:45:26.441314 3798194 net.go:642] Add success. I1121 16:45:26.445258 3798194 net.go:778] primary dev: ETH0 I1121 16:45:26.445275 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:45:26.445289 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:45:29.450409 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:45:29.450493 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:45:29.450517 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:45:36.412823 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:45:36.412848 3798194 memory.go:184] no items to output this cycle I1121 16:45:36.412991 3798194 cpu.go:275] no items to output this cycle E1121 16:45:46.412082 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:45:46.412134 3798194 memory.go:184] no items to output this cycle I1121 16:45:46.412409 3798194 cpu.go:275] no items to output this cycle E1121 16:45:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:45:56.413268 3798194 memory.go:191] Add success. W1121 16:45:56.413296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:45:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:45:56.413311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:45:56.417717 3798194 cpu.go:282] Add success. I1121 16:45:56.443772 3798194 event_worker.go:152] Polling the log file for events... I1121 16:45:56.443990 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:45:56.444584 3798194 net.go:642] Add success. I1121 16:45:56.446200 3798194 net.go:778] primary dev: ETH0 I1121 16:45:56.446214 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:45:56.446228 3798194 net.go:659] Add eni metrics success, count: 7 E1121 16:45:57.450159 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 16:45:57.450175 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:45:57.450329 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:45:57.450379 3798194 disk_worker.go:723] disk space is not compliant W1121 16:45:57.450383 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:45:57.458471 3798194 disk_worker.go:509] system disk:vda1 I1121 16:45:57.458512 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:45:58.208816 3798194 disk_info.go:125] begin check local disk info of client I1121 16:45:58.212399 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:45:58.212411 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000466400 0xc000466440] E1121 16:45:58.447788 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 16:45:58.447801 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 16:45:59.448613 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 16:45:59.449688 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:45:59.449739 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:45:59.449758 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:45:59.471277 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:46:06.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:46:06.412886 3798194 memory.go:184] no items to output this cycle I1121 16:46:06.413035 3798194 cpu.go:275] no items to output this cycle E1121 16:46:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:46:16.411978 3798194 memory.go:184] no items to output this cycle I1121 16:46:16.412133 3798194 cpu.go:275] no items to output this cycle E1121 16:46:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:46:26.411958 3798194 memory.go:191] Add success. I1121 16:46:26.412296 3798194 cpu.go:282] Add success. I1121 16:46:26.442148 3798194 net.go:642] Add success. I1121 16:46:26.446415 3798194 net.go:778] primary dev: ETH0 I1121 16:46:26.446429 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:46:26.446441 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:46:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:46:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:46:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:46:36.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:46:36.411983 3798194 memory.go:184] no items to output this cycle I1121 16:46:36.412145 3798194 cpu.go:275] no items to output this cycle E1121 16:46:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:46:46.411983 3798194 memory.go:184] no items to output this cycle I1121 16:46:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 16:46:56.411231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:46:56.411265 3798194 memory.go:191] Add success. W1121 16:46:56.411290 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:46:56.411302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:46:56.411305 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:46:56.415820 3798194 cpu.go:282] Add success. I1121 16:46:56.441327 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:46:56.441906 3798194 net.go:642] Add success. I1121 16:46:56.443390 3798194 net.go:778] primary dev: ETH0 I1121 16:46:56.443404 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:46:56.443419 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:46:57.450570 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:46:57.450804 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:46:57.450817 3798194 disk_worker.go:723] disk space is not compliant W1121 16:46:57.450820 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:46:57.459656 3798194 disk_worker.go:509] system disk:vda1 I1121 16:46:57.459694 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:46:58.212516 3798194 disk_info.go:125] begin check local disk info of client I1121 16:46:58.216083 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:46:58.216093 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046b4c0 0xc00046b500] I1121 16:46:58.446030 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:46:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:46:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:46:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:46:59.471700 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:47:06.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:47:06.412264 3798194 memory.go:184] no items to output this cycle I1121 16:47:06.412407 3798194 cpu.go:275] no items to output this cycle E1121 16:47:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:47:16.411978 3798194 memory.go:184] no items to output this cycle I1121 16:47:16.412142 3798194 cpu.go:275] no items to output this cycle E1121 16:47:26.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:47:26.411951 3798194 memory.go:191] Add success. I1121 16:47:26.412319 3798194 cpu.go:282] Add success. I1121 16:47:26.441939 3798194 net.go:642] Add success. I1121 16:47:26.446146 3798194 net.go:778] primary dev: ETH0 I1121 16:47:26.446161 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:47:26.446175 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:47:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:47:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:47:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:47:36.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:47:36.411976 3798194 memory.go:184] no items to output this cycle I1121 16:47:36.412129 3798194 cpu.go:275] no items to output this cycle E1121 16:47:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:47:46.411972 3798194 memory.go:184] no items to output this cycle I1121 16:47:46.412139 3798194 cpu.go:275] no items to output this cycle E1121 16:47:56.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:47:56.411963 3798194 memory.go:191] Add success. W1121 16:47:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:47:56.412006 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:47:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:47:56.412366 3798194 cpu.go:282] Add success. I1121 16:47:56.441822 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:47:56.442433 3798194 net.go:642] Add success. I1121 16:47:56.446609 3798194 net.go:778] primary dev: ETH0 I1121 16:47:56.446624 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:47:56.446635 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:47:57.450666 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:47:57.450881 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:47:57.450893 3798194 disk_worker.go:723] disk space is not compliant W1121 16:47:57.450896 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:47:57.459295 3798194 disk_worker.go:509] system disk:vda1 I1121 16:47:57.459338 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:47:58.216194 3798194 disk_info.go:125] begin check local disk info of client I1121 16:47:58.219718 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:47:58.219727 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ec1c0 0xc0003ec200] I1121 16:47:58.446469 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:47:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:47:59.450122 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:47:59.450151 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:47:59.471694 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:48:06.412771 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:48:06.412794 3798194 memory.go:184] no items to output this cycle I1121 16:48:06.412949 3798194 cpu.go:275] no items to output this cycle E1121 16:48:16.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:48:16.411983 3798194 memory.go:184] no items to output this cycle I1121 16:48:16.412151 3798194 cpu.go:275] no items to output this cycle I1121 16:48:17.515330 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:48:17.515339 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:48:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:48:26.413265 3798194 memory.go:191] Add success. I1121 16:48:26.417329 3798194 cpu.go:282] Add success. I1121 16:48:26.442138 3798194 net.go:642] Add success. I1121 16:48:26.443790 3798194 net.go:778] primary dev: ETH0 I1121 16:48:26.443806 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:48:26.443819 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:48:29.449995 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:48:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:48:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:48:36.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:48:36.411972 3798194 memory.go:184] no items to output this cycle I1121 16:48:36.412140 3798194 cpu.go:275] no items to output this cycle E1121 16:48:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:48:46.411977 3798194 memory.go:184] no items to output this cycle I1121 16:48:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 16:48:56.413285 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:48:56.413332 3798194 memory.go:191] Add success. W1121 16:48:56.413364 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:48:56.413380 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:48:56.413384 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:48:56.417863 3798194 cpu.go:282] Add success. I1121 16:48:56.441355 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:48:56.441826 3798194 net.go:642] Add success. I1121 16:48:56.443289 3798194 net.go:778] primary dev: ETH0 I1121 16:48:56.443304 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:48:56.443316 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:48:57.450291 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:48:57.450493 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:48:57.450504 3798194 disk_worker.go:723] disk space is not compliant W1121 16:48:57.450507 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:48:57.458530 3798194 disk_worker.go:509] system disk:vda1 I1121 16:48:57.458570 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:48:58.223148 3798194 disk_info.go:125] begin check local disk info of client I1121 16:48:58.226668 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:48:58.226677 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000269c80 0xc000269cc0] I1121 16:48:58.445933 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:48:59.450035 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:48:59.450129 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:48:59.450158 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:48:59.471669 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:49:06.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:49:06.412253 3798194 memory.go:184] no items to output this cycle I1121 16:49:06.412403 3798194 cpu.go:275] no items to output this cycle E1121 16:49:16.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:49:16.411988 3798194 memory.go:184] no items to output this cycle I1121 16:49:16.412149 3798194 cpu.go:275] no items to output this cycle E1121 16:49:26.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:49:26.411945 3798194 memory.go:191] Add success. I1121 16:49:26.412311 3798194 cpu.go:282] Add success. I1121 16:49:26.441919 3798194 net.go:642] Add success. I1121 16:49:26.445793 3798194 net.go:778] primary dev: ETH0 I1121 16:49:26.445808 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:49:26.445821 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:49:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:49:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:49:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:49:36.412088 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:49:36.412124 3798194 memory.go:184] no items to output this cycle I1121 16:49:36.412272 3798194 cpu.go:275] no items to output this cycle E1121 16:49:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:49:46.411985 3798194 memory.go:184] no items to output this cycle I1121 16:49:46.412140 3798194 cpu.go:275] no items to output this cycle E1121 16:49:56.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:49:56.412277 3798194 memory.go:191] Add success. W1121 16:49:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:49:56.412322 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:49:56.412325 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:49:56.416758 3798194 cpu.go:282] Add success. I1121 16:49:56.440966 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:49:56.441567 3798194 net.go:642] Add success. I1121 16:49:56.443146 3798194 net.go:778] primary dev: ETH0 I1121 16:49:56.443160 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:49:56.443173 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:49:57.450789 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:49:57.450958 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:49:57.451013 3798194 disk_worker.go:723] disk space is not compliant W1121 16:49:57.451016 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:49:57.460347 3798194 disk_worker.go:509] system disk:vda1 I1121 16:49:57.460390 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:49:58.226778 3798194 disk_info.go:125] begin check local disk info of client I1121 16:49:58.230337 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:49:58.230347 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030f500 0xc00030f540] I1121 16:49:58.446386 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:49:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:49:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:49:59.450504 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:49:59.471024 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:50:06.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:50:06.413265 3798194 memory.go:184] no items to output this cycle I1121 16:50:06.413414 3798194 cpu.go:275] no items to output this cycle E1121 16:50:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:50:16.411980 3798194 memory.go:184] no items to output this cycle I1121 16:50:16.412139 3798194 cpu.go:275] no items to output this cycle E1121 16:50:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:50:26.413265 3798194 memory.go:191] Add success. I1121 16:50:26.417644 3798194 cpu.go:282] Add success. I1121 16:50:26.441254 3798194 net.go:642] Add success. I1121 16:50:26.442724 3798194 net.go:778] primary dev: ETH0 I1121 16:50:26.442740 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:50:26.442753 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:50:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:50:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:50:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:50:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:50:36.412250 3798194 memory.go:184] no items to output this cycle I1121 16:50:36.412396 3798194 cpu.go:275] no items to output this cycle E1121 16:50:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:50:46.411980 3798194 memory.go:184] no items to output this cycle I1121 16:50:46.412135 3798194 cpu.go:275] no items to output this cycle E1121 16:50:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:50:56.411949 3798194 memory.go:191] Add success. W1121 16:50:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:50:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:50:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:50:56.412340 3798194 cpu.go:282] Add success. I1121 16:50:56.444475 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:50:56.444966 3798194 net.go:642] Add success. I1121 16:50:56.448973 3798194 net.go:778] primary dev: ETH0 I1121 16:50:56.448988 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:50:56.449005 3798194 net.go:659] Add eni metrics success, count: 7 E1121 16:50:57.450311 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 16:50:57.450325 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:50:57.450516 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:50:57.450525 3798194 disk_worker.go:723] disk space is not compliant W1121 16:50:57.450528 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:50:57.458571 3798194 disk_worker.go:509] system disk:vda1 I1121 16:50:57.458612 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:50:58.233418 3798194 disk_info.go:125] begin check local disk info of client I1121 16:50:58.236988 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:50:58.236997 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ffb80 0xc0003ffbc0] E1121 16:50:58.447758 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 16:50:58.447773 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 16:50:59.448626 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 16:50:59.449704 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:50:59.449755 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:50:59.449774 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:50:59.470732 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:51:06.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:51:06.412267 3798194 memory.go:184] no items to output this cycle I1121 16:51:06.412413 3798194 cpu.go:275] no items to output this cycle E1121 16:51:16.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:51:16.413258 3798194 memory.go:184] no items to output this cycle I1121 16:51:16.413403 3798194 cpu.go:275] no items to output this cycle I1121 16:51:17.516426 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:51:17.516435 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:51:26.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:51:26.412881 3798194 memory.go:191] Add success. I1121 16:51:26.417262 3798194 cpu.go:282] Add success. I1121 16:51:26.441730 3798194 net.go:642] Add success. I1121 16:51:26.443136 3798194 net.go:778] primary dev: ETH0 I1121 16:51:26.443151 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:51:26.443165 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:51:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:51:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:51:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:51:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:51:36.411981 3798194 memory.go:184] no items to output this cycle I1121 16:51:36.412138 3798194 cpu.go:275] no items to output this cycle E1121 16:51:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:51:46.411981 3798194 memory.go:184] no items to output this cycle I1121 16:51:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 16:51:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:51:56.411953 3798194 memory.go:191] Add success. W1121 16:51:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:51:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:51:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:51:56.412355 3798194 cpu.go:282] Add success. I1121 16:51:56.441805 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:51:56.442382 3798194 net.go:642] Add success. I1121 16:51:56.446468 3798194 net.go:778] primary dev: ETH0 I1121 16:51:56.446484 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:51:56.446495 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:51:57.450321 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:51:57.450482 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:51:57.450535 3798194 disk_worker.go:723] disk space is not compliant W1121 16:51:57.450538 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:51:57.458763 3798194 disk_worker.go:509] system disk:vda1 I1121 16:51:57.458810 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:51:58.237117 3798194 disk_info.go:125] begin check local disk info of client I1121 16:51:58.240661 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:51:58.240670 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037f300 0xc00037f340] I1121 16:51:58.445698 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:51:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:51:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:51:59.450144 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:51:59.471672 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:52:06.412093 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:52:06.412129 3798194 memory.go:184] no items to output this cycle I1121 16:52:06.412275 3798194 cpu.go:275] no items to output this cycle E1121 16:52:16.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:52:16.412258 3798194 memory.go:184] no items to output this cycle I1121 16:52:16.412400 3798194 cpu.go:275] no items to output this cycle E1121 16:52:26.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:52:26.413262 3798194 memory.go:191] Add success. I1121 16:52:26.417613 3798194 cpu.go:282] Add success. I1121 16:52:26.442275 3798194 net.go:642] Add success. I1121 16:52:26.443718 3798194 net.go:778] primary dev: ETH0 I1121 16:52:26.443732 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:52:26.443748 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:52:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:52:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:52:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:52:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:52:36.412254 3798194 memory.go:184] no items to output this cycle I1121 16:52:36.412397 3798194 cpu.go:275] no items to output this cycle E1121 16:52:46.412251 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:52:46.412277 3798194 memory.go:184] no items to output this cycle I1121 16:52:46.412425 3798194 cpu.go:275] no items to output this cycle E1121 16:52:56.411930 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:52:56.411963 3798194 memory.go:191] Add success. W1121 16:52:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:52:56.412007 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:52:56.412010 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:52:56.412332 3798194 cpu.go:282] Add success. I1121 16:52:56.441933 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:52:56.442445 3798194 net.go:642] Add success. I1121 16:52:56.446253 3798194 net.go:778] primary dev: ETH0 I1121 16:52:56.446267 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:52:56.446280 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:52:57.451882 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:52:57.452088 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:52:57.452098 3798194 disk_worker.go:723] disk space is not compliant W1121 16:52:57.452102 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:52:57.464210 3798194 disk_worker.go:509] system disk:vda1 I1121 16:52:57.464261 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:52:58.243145 3798194 disk_info.go:125] begin check local disk info of client I1121 16:52:58.246726 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:52:58.246736 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037edc0 0xc00037ee00] I1121 16:52:58.445704 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:52:59.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:52:59.450122 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:52:59.450149 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:52:59.471646 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:53:06.412398 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:53:06.412423 3798194 memory.go:184] no items to output this cycle I1121 16:53:06.412570 3798194 cpu.go:275] no items to output this cycle E1121 16:53:16.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:53:16.412252 3798194 memory.go:184] no items to output this cycle I1121 16:53:16.412402 3798194 cpu.go:275] no items to output this cycle E1121 16:53:26.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:53:26.412267 3798194 memory.go:191] Add success. I1121 16:53:26.416297 3798194 cpu.go:282] Add success. I1121 16:53:26.441739 3798194 net.go:642] Add success. I1121 16:53:26.443140 3798194 net.go:778] primary dev: ETH0 I1121 16:53:26.443154 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:53:26.443167 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:53:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:53:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:53:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:53:36.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:53:36.412278 3798194 memory.go:184] no items to output this cycle I1121 16:53:36.412420 3798194 cpu.go:275] no items to output this cycle E1121 16:53:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:53:46.411971 3798194 memory.go:184] no items to output this cycle I1121 16:53:46.412124 3798194 cpu.go:275] no items to output this cycle E1121 16:53:56.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:53:56.412265 3798194 memory.go:191] Add success. W1121 16:53:56.412293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:53:56.412304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:53:56.412308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:53:56.416585 3798194 cpu.go:282] Add success. I1121 16:53:56.440925 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:53:56.441693 3798194 net.go:642] Add success. I1121 16:53:56.443828 3798194 net.go:778] primary dev: ETH0 I1121 16:53:56.443842 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:53:56.443853 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:53:57.448879 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:53:57.449082 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:53:57.449159 3798194 disk_worker.go:723] disk space is not compliant W1121 16:53:57.449163 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:53:57.450938 3798194 disk_worker.go:509] system disk:vda1 I1121 16:53:57.450977 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:53:58.249146 3798194 disk_info.go:125] begin check local disk info of client I1121 16:53:58.252741 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:53:58.252751 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00049eb00 0xc00049eb40] I1121 16:53:58.446432 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:53:59.450386 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:53:59.450466 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:53:59.450491 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:53:59.471013 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:54:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:54:06.412254 3798194 memory.go:184] no items to output this cycle I1121 16:54:06.412394 3798194 cpu.go:275] no items to output this cycle E1121 16:54:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:54:16.413261 3798194 memory.go:184] no items to output this cycle I1121 16:54:16.413412 3798194 cpu.go:275] no items to output this cycle I1121 16:54:17.516596 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:54:17.516607 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:54:26.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:54:26.412265 3798194 memory.go:191] Add success. I1121 16:54:26.416469 3798194 cpu.go:282] Add success. I1121 16:54:26.441994 3798194 net.go:642] Add success. I1121 16:54:26.443414 3798194 net.go:778] primary dev: ETH0 I1121 16:54:26.443429 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:54:26.443441 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:54:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:54:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:54:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:54:36.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:54:36.412873 3798194 memory.go:184] no items to output this cycle I1121 16:54:36.413021 3798194 cpu.go:275] no items to output this cycle E1121 16:54:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:54:46.411974 3798194 memory.go:184] no items to output this cycle I1121 16:54:46.412134 3798194 cpu.go:275] no items to output this cycle E1121 16:54:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:54:56.411953 3798194 memory.go:191] Add success. W1121 16:54:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:54:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:54:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:54:56.412333 3798194 cpu.go:282] Add success. I1121 16:54:56.441893 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:54:56.442450 3798194 net.go:642] Add success. I1121 16:54:56.446668 3798194 net.go:778] primary dev: ETH0 I1121 16:54:56.446684 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:54:56.446697 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:54:57.450723 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:54:57.450883 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:54:57.450932 3798194 disk_worker.go:723] disk space is not compliant W1121 16:54:57.450936 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:54:57.459050 3798194 disk_worker.go:509] system disk:vda1 I1121 16:54:57.459096 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:54:58.252847 3798194 disk_info.go:125] begin check local disk info of client I1121 16:54:58.256352 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:54:58.256361 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ebec0 0xc0003ebf00] I1121 16:54:58.446409 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:54:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:54:59.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:54:59.450146 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:54:59.471696 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:55:06.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:55:06.412266 3798194 memory.go:184] no items to output this cycle I1121 16:55:06.412409 3798194 cpu.go:275] no items to output this cycle E1121 16:55:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:55:16.411979 3798194 memory.go:184] no items to output this cycle I1121 16:55:16.412137 3798194 cpu.go:275] no items to output this cycle E1121 16:55:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:55:26.411959 3798194 memory.go:191] Add success. I1121 16:55:26.412326 3798194 cpu.go:282] Add success. I1121 16:55:26.441978 3798194 net.go:642] Add success. I1121 16:55:26.446049 3798194 net.go:778] primary dev: ETH0 I1121 16:55:26.446065 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:55:26.446079 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:55:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:55:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:55:29.450147 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:55:36.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:55:36.412262 3798194 memory.go:184] no items to output this cycle I1121 16:55:36.412410 3798194 cpu.go:275] no items to output this cycle E1121 16:55:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:55:46.411986 3798194 memory.go:184] no items to output this cycle I1121 16:55:46.412147 3798194 cpu.go:275] no items to output this cycle E1121 16:55:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:55:56.411956 3798194 memory.go:191] Add success. W1121 16:55:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:55:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:55:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:55:56.412351 3798194 cpu.go:282] Add success. I1121 16:55:56.441558 3798194 event_worker.go:152] Polling the log file for events... I1121 16:55:56.444405 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:55:56.444885 3798194 net.go:642] Add success. I1121 16:55:56.449139 3798194 net.go:778] primary dev: ETH0 I1121 16:55:56.449155 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:55:56.449168 3798194 net.go:659] Add eni metrics success, count: 7 E1121 16:55:57.450384 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 16:55:57.450398 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:55:57.450559 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:55:57.450612 3798194 disk_worker.go:723] disk space is not compliant W1121 16:55:57.450616 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:55:57.458807 3798194 disk_worker.go:509] system disk:vda1 I1121 16:55:57.458847 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:55:58.256463 3798194 disk_info.go:125] begin check local disk info of client I1121 16:55:58.260044 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:55:58.260053 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000313380 0xc0003133c0] E1121 16:55:58.447766 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 16:55:58.447781 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 16:55:59.448608 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 16:55:59.449683 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:55:59.449733 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:55:59.449760 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:55:59.471267 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:56:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:56:06.411983 3798194 memory.go:184] no items to output this cycle I1121 16:56:06.412145 3798194 cpu.go:275] no items to output this cycle E1121 16:56:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:56:16.411978 3798194 memory.go:184] no items to output this cycle I1121 16:56:16.412133 3798194 cpu.go:275] no items to output this cycle E1121 16:56:26.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:56:26.412896 3798194 memory.go:191] Add success. I1121 16:56:26.422502 3798194 cpu.go:282] Add success. I1121 16:56:26.442285 3798194 net.go:642] Add success. I1121 16:56:26.443714 3798194 net.go:778] primary dev: ETH0 I1121 16:56:26.443730 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:56:26.443744 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:56:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:56:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:56:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:56:36.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:56:36.412255 3798194 memory.go:184] no items to output this cycle I1121 16:56:36.412402 3798194 cpu.go:275] no items to output this cycle E1121 16:56:46.412250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:56:46.412275 3798194 memory.go:184] no items to output this cycle I1121 16:56:46.412423 3798194 cpu.go:275] no items to output this cycle E1121 16:56:56.412098 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:56:56.412318 3798194 memory.go:191] Add success. W1121 16:56:56.412379 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:56:56.412401 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:56:56.412407 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:56:56.413527 3798194 cpu.go:282] Add success. I1121 16:56:56.446510 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:56:56.447441 3798194 net.go:642] Add success. I1121 16:56:56.452145 3798194 net.go:778] primary dev: ETH0 I1121 16:56:56.452169 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:56:56.452190 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:56:57.445312 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:56:57.445590 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:56:57.445602 3798194 disk_worker.go:723] disk space is not compliant W1121 16:56:57.445606 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:56:57.448835 3798194 disk_worker.go:509] system disk:vda1 I1121 16:56:57.448878 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:56:58.260158 3798194 disk_info.go:125] begin check local disk info of client I1121 16:56:58.263740 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:56:58.263749 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ed00 0xc00030ed40] I1121 16:56:58.446179 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:56:59.450514 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:56:59.450600 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:56:59.450626 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:56:59.471129 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:57:06.411971 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:57:06.411995 3798194 memory.go:184] no items to output this cycle I1121 16:57:06.412162 3798194 cpu.go:275] no items to output this cycle E1121 16:57:16.411971 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:57:16.411991 3798194 memory.go:184] no items to output this cycle I1121 16:57:16.412160 3798194 cpu.go:275] no items to output this cycle I1121 16:57:17.517569 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 16:57:17.517578 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 16:57:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:57:26.413266 3798194 memory.go:191] Add success. I1121 16:57:26.417347 3798194 cpu.go:282] Add success. I1121 16:57:26.441902 3798194 net.go:642] Add success. I1121 16:57:26.443360 3798194 net.go:778] primary dev: ETH0 I1121 16:57:26.443374 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:57:26.443386 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:57:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:57:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:57:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:57:36.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:57:36.412258 3798194 memory.go:184] no items to output this cycle I1121 16:57:36.412397 3798194 cpu.go:275] no items to output this cycle E1121 16:57:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:57:46.411980 3798194 memory.go:184] no items to output this cycle I1121 16:57:46.412137 3798194 cpu.go:275] no items to output this cycle E1121 16:57:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:57:56.411957 3798194 memory.go:191] Add success. W1121 16:57:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:57:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:57:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:57:56.412353 3798194 cpu.go:282] Add success. I1121 16:57:56.441822 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:57:56.442429 3798194 net.go:642] Add success. I1121 16:57:56.446962 3798194 net.go:778] primary dev: ETH0 I1121 16:57:56.446978 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:57:56.446991 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:57:57.450271 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:57:57.450434 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:57:57.450490 3798194 disk_worker.go:723] disk space is not compliant W1121 16:57:57.450493 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:57:57.458563 3798194 disk_worker.go:509] system disk:vda1 I1121 16:57:57.458605 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:57:58.263851 3798194 disk_info.go:125] begin check local disk info of client I1121 16:57:58.267466 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:57:58.267474 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003700c0 0xc000370100] I1121 16:57:58.445996 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:57:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:57:59.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:57:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:57:59.471673 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:58:06.412759 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:58:06.412781 3798194 memory.go:184] no items to output this cycle I1121 16:58:06.412935 3798194 cpu.go:275] no items to output this cycle E1121 16:58:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:58:16.411977 3798194 memory.go:184] no items to output this cycle I1121 16:58:16.412138 3798194 cpu.go:275] no items to output this cycle E1121 16:58:26.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:58:26.411958 3798194 memory.go:191] Add success. I1121 16:58:26.412322 3798194 cpu.go:282] Add success. I1121 16:58:26.442042 3798194 net.go:642] Add success. I1121 16:58:26.445821 3798194 net.go:778] primary dev: ETH0 I1121 16:58:26.445838 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:58:26.445851 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:58:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:58:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:58:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:58:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:58:36.411980 3798194 memory.go:184] no items to output this cycle I1121 16:58:36.412133 3798194 cpu.go:275] no items to output this cycle E1121 16:58:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:58:46.412255 3798194 memory.go:184] no items to output this cycle I1121 16:58:46.412408 3798194 cpu.go:275] no items to output this cycle E1121 16:58:56.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:58:56.412894 3798194 memory.go:191] Add success. W1121 16:58:56.412927 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:58:56.412938 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:58:56.412942 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:58:56.416957 3798194 cpu.go:282] Add success. I1121 16:58:56.441453 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:58:56.442024 3798194 net.go:642] Add success. I1121 16:58:56.443682 3798194 net.go:778] primary dev: ETH0 I1121 16:58:56.443698 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:58:56.443711 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:58:57.450751 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:58:57.450961 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:58:57.450973 3798194 disk_worker.go:723] disk space is not compliant W1121 16:58:57.450976 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:58:57.459293 3798194 disk_worker.go:509] system disk:vda1 I1121 16:58:57.459338 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:58:58.267572 3798194 disk_info.go:125] begin check local disk info of client I1121 16:58:58.271106 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:58:58.271140 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003bd780 0xc0003bd7c0] I1121 16:58:58.445997 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:58:59.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:58:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:58:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:58:59.471670 3798194 disk_local_worker.go:436] Get disk info: [] E1121 16:59:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:59:06.411980 3798194 memory.go:184] no items to output this cycle I1121 16:59:06.412134 3798194 cpu.go:275] no items to output this cycle E1121 16:59:16.412223 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:59:16.412246 3798194 memory.go:184] no items to output this cycle I1121 16:59:16.412390 3798194 cpu.go:275] no items to output this cycle E1121 16:59:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:59:26.411951 3798194 memory.go:191] Add success. I1121 16:59:26.412317 3798194 cpu.go:282] Add success. I1121 16:59:26.442332 3798194 net.go:642] Add success. I1121 16:59:26.446616 3798194 net.go:778] primary dev: ETH0 I1121 16:59:26.446633 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:59:26.446655 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:59:29.450232 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:59:29.450325 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:59:29.450355 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 16:59:36.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:59:36.411991 3798194 memory.go:184] no items to output this cycle I1121 16:59:36.412149 3798194 cpu.go:275] no items to output this cycle E1121 16:59:46.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:59:46.411992 3798194 memory.go:184] no items to output this cycle I1121 16:59:46.412156 3798194 cpu.go:275] no items to output this cycle E1121 16:59:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 16:59:56.411956 3798194 memory.go:191] Add success. W1121 16:59:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 16:59:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 16:59:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 16:59:56.412340 3798194 cpu.go:282] Add success. I1121 16:59:56.441831 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 16:59:56.442360 3798194 net.go:642] Add success. I1121 16:59:56.446244 3798194 net.go:778] primary dev: ETH0 I1121 16:59:56.446260 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 16:59:56.446273 3798194 net.go:659] Add eni metrics success, count: 7 I1121 16:59:57.450366 3798194 custom_config.go:106] query custom config with name: gpu W1121 16:59:57.450534 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 16:59:57.450590 3798194 disk_worker.go:723] disk space is not compliant W1121 16:59:57.450594 3798194 disk_worker.go:743] disk inode is not compliant I1121 16:59:57.458717 3798194 disk_worker.go:509] system disk:vda1 I1121 16:59:57.458759 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 16:59:58.271247 3798194 disk_info.go:125] begin check local disk info of client I1121 16:59:58.274844 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 16:59:58.274853 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00042dc00 0xc00042dc40] I1121 16:59:58.446444 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 16:59:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 16:59:59.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 16:59:59.450510 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 16:59:59.471042 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:00:06.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:00:06.413260 3798194 memory.go:184] no items to output this cycle I1121 17:00:06.413408 3798194 cpu.go:275] no items to output this cycle E1121 17:00:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:00:16.411978 3798194 memory.go:184] no items to output this cycle I1121 17:00:16.412133 3798194 cpu.go:275] no items to output this cycle I1121 17:00:17.518261 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:00:17.518270 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:00:26.411228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:00:26.411265 3798194 memory.go:191] Add success. I1121 17:00:26.411603 3798194 cpu.go:282] Add success. I1121 17:00:26.442139 3798194 net.go:642] Add success. I1121 17:00:26.446672 3798194 net.go:778] primary dev: ETH0 I1121 17:00:26.446689 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:00:26.446702 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:00:29.450029 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:00:29.450137 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:00:29.450165 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:00:36.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:00:36.413257 3798194 memory.go:184] no items to output this cycle I1121 17:00:36.413398 3798194 cpu.go:275] no items to output this cycle E1121 17:00:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:00:46.411975 3798194 memory.go:184] no items to output this cycle I1121 17:00:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 17:00:56.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:00:56.413266 3798194 memory.go:191] Add success. W1121 17:00:56.413293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:00:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:00:56.413308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:00:56.418389 3798194 cpu.go:282] Add success. I1121 17:00:56.442859 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:00:56.443908 3798194 net.go:642] Add success. I1121 17:00:56.447367 3798194 net.go:778] primary dev: ETH0 I1121 17:00:56.447382 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:00:56.447394 3798194 net.go:659] Add eni metrics success, count: 7 E1121 17:00:57.447986 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 17:00:57.448009 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:00:57.448252 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:00:57.448262 3798194 disk_worker.go:723] disk space is not compliant W1121 17:00:57.448265 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:00:57.451142 3798194 disk_worker.go:509] system disk:vda1 I1121 17:00:57.451180 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:00:58.274961 3798194 disk_info.go:125] begin check local disk info of client I1121 17:00:58.278664 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:00:58.278673 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004c1980 0xc0004c19c0] E1121 17:00:58.448178 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 17:00:58.448192 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 17:00:59.448979 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 17:00:59.450053 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:00:59.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:00:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:00:59.471644 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:01:06.413253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:01:06.413276 3798194 memory.go:184] no items to output this cycle I1121 17:01:06.413420 3798194 cpu.go:275] no items to output this cycle E1121 17:01:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:01:16.411984 3798194 memory.go:184] no items to output this cycle I1121 17:01:16.412143 3798194 cpu.go:275] no items to output this cycle E1121 17:01:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:01:26.411953 3798194 memory.go:191] Add success. I1121 17:01:26.412322 3798194 cpu.go:282] Add success. I1121 17:01:26.442107 3798194 net.go:642] Add success. I1121 17:01:26.446071 3798194 net.go:778] primary dev: ETH0 I1121 17:01:26.446087 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:01:26.446099 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:01:29.450036 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:01:29.450133 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:01:29.450158 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:01:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:01:36.411981 3798194 memory.go:184] no items to output this cycle I1121 17:01:36.412132 3798194 cpu.go:275] no items to output this cycle E1121 17:01:46.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:01:46.412255 3798194 memory.go:184] no items to output this cycle I1121 17:01:46.412400 3798194 cpu.go:275] no items to output this cycle E1121 17:01:56.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:01:56.413280 3798194 memory.go:191] Add success. W1121 17:01:56.413309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:01:56.413321 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:01:56.413326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:01:56.417477 3798194 cpu.go:282] Add success. I1121 17:01:56.441894 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:01:56.442468 3798194 net.go:642] Add success. I1121 17:01:56.444106 3798194 net.go:778] primary dev: ETH0 I1121 17:01:56.444134 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:01:56.444148 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:01:57.450365 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:01:57.450529 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:01:57.450580 3798194 disk_worker.go:723] disk space is not compliant W1121 17:01:57.450583 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:01:57.458785 3798194 disk_worker.go:509] system disk:vda1 I1121 17:01:57.458830 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:01:58.282140 3798194 disk_info.go:125] begin check local disk info of client I1121 17:01:58.285732 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:01:58.285741 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004409c0 0xc000440a00] I1121 17:01:58.446453 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:01:59.450560 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:01:59.450655 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:01:59.450684 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:01:59.470728 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:02:06.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:02:06.411990 3798194 memory.go:184] no items to output this cycle I1121 17:02:06.412145 3798194 cpu.go:275] no items to output this cycle E1121 17:02:16.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:02:16.411989 3798194 memory.go:184] no items to output this cycle I1121 17:02:16.412155 3798194 cpu.go:275] no items to output this cycle E1121 17:02:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:02:26.411951 3798194 memory.go:191] Add success. I1121 17:02:26.412335 3798194 cpu.go:282] Add success. I1121 17:02:26.441930 3798194 net.go:642] Add success. I1121 17:02:26.445951 3798194 net.go:778] primary dev: ETH0 I1121 17:02:26.445968 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:02:26.445990 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:02:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:02:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:02:29.450147 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:02:36.412753 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:02:36.412777 3798194 memory.go:184] no items to output this cycle I1121 17:02:36.412926 3798194 cpu.go:275] no items to output this cycle E1121 17:02:46.412865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:02:46.412887 3798194 memory.go:184] no items to output this cycle I1121 17:02:46.413041 3798194 cpu.go:275] no items to output this cycle E1121 17:02:56.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:02:56.412270 3798194 memory.go:191] Add success. W1121 17:02:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:02:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:02:56.412312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:02:56.416653 3798194 cpu.go:282] Add success. I1121 17:02:56.440952 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:02:56.441543 3798194 net.go:642] Add success. I1121 17:02:56.443121 3798194 net.go:778] primary dev: ETH0 I1121 17:02:56.443135 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:02:56.443149 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:02:57.450251 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:02:57.450449 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:02:57.450460 3798194 disk_worker.go:723] disk space is not compliant W1121 17:02:57.450464 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:02:57.458464 3798194 disk_worker.go:509] system disk:vda1 I1121 17:02:57.458506 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:02:58.285847 3798194 disk_info.go:125] begin check local disk info of client I1121 17:02:58.289443 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:02:58.289453 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ab2c0 0xc0003ab300] I1121 17:02:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:02:59.450106 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:02:59.450191 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:02:59.450218 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:02:59.470684 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:03:06.412250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:03:06.412272 3798194 memory.go:184] no items to output this cycle I1121 17:03:06.412420 3798194 cpu.go:275] no items to output this cycle E1121 17:03:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:03:16.413259 3798194 memory.go:184] no items to output this cycle I1121 17:03:16.413402 3798194 cpu.go:275] no items to output this cycle I1121 17:03:17.519272 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:03:17.519280 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:03:26.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:03:26.412264 3798194 memory.go:191] Add success. I1121 17:03:26.416938 3798194 cpu.go:282] Add success. I1121 17:03:26.441502 3798194 net.go:642] Add success. I1121 17:03:26.442958 3798194 net.go:778] primary dev: ETH0 I1121 17:03:26.442974 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:03:26.442988 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:03:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:03:29.450459 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:03:29.450488 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:03:36.411231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:03:36.411256 3798194 memory.go:184] no items to output this cycle I1121 17:03:36.411393 3798194 cpu.go:275] no items to output this cycle E1121 17:03:46.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:03:46.411989 3798194 memory.go:184] no items to output this cycle I1121 17:03:46.412151 3798194 cpu.go:275] no items to output this cycle E1121 17:03:56.412281 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:03:56.412328 3798194 memory.go:191] Add success. W1121 17:03:56.412372 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:03:56.412390 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:03:56.412395 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:03:56.425476 3798194 cpu.go:282] Add success. I1121 17:03:56.444055 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:03:56.444938 3798194 net.go:642] Add success. I1121 17:03:56.446724 3798194 net.go:778] primary dev: ETH0 I1121 17:03:56.446745 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:03:56.446762 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:03:57.445305 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:03:57.445519 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:03:57.445605 3798194 disk_worker.go:723] disk space is not compliant W1121 17:03:57.445608 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:03:57.448176 3798194 disk_worker.go:509] system disk:vda1 I1121 17:03:57.448216 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:03:58.292145 3798194 disk_info.go:125] begin check local disk info of client I1121 17:03:58.295827 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:03:58.295835 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003aadc0 0xc0003aae00] I1121 17:03:58.446647 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:03:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:03:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:03:59.450151 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:03:59.470674 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:04:06.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:04:06.411990 3798194 memory.go:184] no items to output this cycle I1121 17:04:06.412159 3798194 cpu.go:275] no items to output this cycle E1121 17:04:16.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:04:16.412259 3798194 memory.go:184] no items to output this cycle I1121 17:04:16.412405 3798194 cpu.go:275] no items to output this cycle E1121 17:04:26.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:04:26.412270 3798194 memory.go:191] Add success. I1121 17:04:26.416809 3798194 cpu.go:282] Add success. I1121 17:04:26.441600 3798194 net.go:642] Add success. I1121 17:04:26.443205 3798194 net.go:778] primary dev: ETH0 I1121 17:04:26.443220 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:04:26.443232 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:04:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:04:29.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:04:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:04:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:04:36.411978 3798194 memory.go:184] no items to output this cycle I1121 17:04:36.412142 3798194 cpu.go:275] no items to output this cycle E1121 17:04:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:04:46.411982 3798194 memory.go:184] no items to output this cycle I1121 17:04:46.412137 3798194 cpu.go:275] no items to output this cycle E1121 17:04:56.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:04:56.413271 3798194 memory.go:191] Add success. W1121 17:04:56.413302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:04:56.413314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:04:56.413318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:04:56.417702 3798194 cpu.go:282] Add success. I1121 17:04:56.441143 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:04:56.441934 3798194 net.go:642] Add success. I1121 17:04:56.443454 3798194 net.go:778] primary dev: ETH0 I1121 17:04:56.443470 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:04:56.443483 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:04:57.450793 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:04:57.450997 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:04:57.451022 3798194 disk_worker.go:723] disk space is not compliant W1121 17:04:57.451025 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:04:57.459229 3798194 disk_worker.go:509] system disk:vda1 I1121 17:04:57.459271 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:04:58.299146 3798194 disk_info.go:125] begin check local disk info of client I1121 17:04:58.302725 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:04:58.302735 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004c0b00 0xc0004c0b40] I1121 17:04:58.446610 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:04:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:04:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:04:59.450147 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:04:59.471671 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:05:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:05:06.412256 3798194 memory.go:184] no items to output this cycle I1121 17:05:06.412404 3798194 cpu.go:275] no items to output this cycle E1121 17:05:16.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:05:16.411988 3798194 memory.go:184] no items to output this cycle I1121 17:05:16.412142 3798194 cpu.go:275] no items to output this cycle E1121 17:05:26.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:05:26.411944 3798194 memory.go:191] Add success. I1121 17:05:26.412310 3798194 cpu.go:282] Add success. I1121 17:05:26.442122 3798194 net.go:642] Add success. I1121 17:05:26.446055 3798194 net.go:778] primary dev: ETH0 I1121 17:05:26.446071 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:05:26.446083 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:05:29.450409 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:05:29.450487 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:05:29.450515 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:05:36.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:05:36.413264 3798194 memory.go:184] no items to output this cycle I1121 17:05:36.413406 3798194 cpu.go:275] no items to output this cycle E1121 17:05:46.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:05:46.413258 3798194 memory.go:184] no items to output this cycle I1121 17:05:46.413403 3798194 cpu.go:275] no items to output this cycle E1121 17:05:56.413250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:05:56.413285 3798194 memory.go:191] Add success. W1121 17:05:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:05:56.413323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:05:56.413327 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:05:56.417558 3798194 cpu.go:282] Add success. I1121 17:05:56.441686 3798194 event_worker.go:152] Polling the log file for events... I1121 17:05:56.444595 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:05:56.445052 3798194 net.go:642] Add success. I1121 17:05:56.446427 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795250156, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 17:05:56.446550 3798194 net.go:778] primary dev: ETH0 I1121 17:05:56.446564 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:05:56.446576 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:05:56.457570 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 17:05:57.448902 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 17:05:57.448917 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:05:57.449157 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:05:57.449168 3798194 disk_worker.go:723] disk space is not compliant W1121 17:05:57.449171 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:05:57.450949 3798194 disk_worker.go:509] system disk:vda1 I1121 17:05:57.450990 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:05:58.302833 3798194 disk_info.go:125] begin check local disk info of client I1121 17:05:58.306474 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:05:58.306484 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041eac0 0xc00041eb00] E1121 17:05:58.447793 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 17:05:58.447808 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 17:05:59.448981 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 17:05:59.450056 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:05:59.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:05:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:05:59.471709 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:06:06.411236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:06:06.411258 3798194 memory.go:184] no items to output this cycle I1121 17:06:06.411402 3798194 cpu.go:275] no items to output this cycle E1121 17:06:16.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:06:16.413276 3798194 memory.go:184] no items to output this cycle I1121 17:06:16.413435 3798194 cpu.go:275] no items to output this cycle I1121 17:06:17.519440 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:06:17.519448 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:06:26.413281 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:06:26.413316 3798194 memory.go:191] Add success. I1121 17:06:26.417734 3798194 cpu.go:282] Add success. I1121 17:06:26.442242 3798194 net.go:642] Add success. I1121 17:06:26.443733 3798194 net.go:778] primary dev: ETH0 I1121 17:06:26.443748 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:06:26.443761 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:06:29.450548 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:06:29.450627 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:06:29.450657 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:06:36.411708 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:06:36.411731 3798194 memory.go:184] no items to output this cycle I1121 17:06:36.411895 3798194 cpu.go:275] no items to output this cycle E1121 17:06:46.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:06:46.411969 3798194 memory.go:184] no items to output this cycle I1121 17:06:46.412126 3798194 cpu.go:275] no items to output this cycle E1121 17:06:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:06:56.412273 3798194 memory.go:191] Add success. W1121 17:06:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:06:56.412314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:06:56.412318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:06:56.416750 3798194 cpu.go:282] Add success. I1121 17:06:56.441159 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:06:56.441648 3798194 net.go:642] Add success. I1121 17:06:56.443167 3798194 net.go:778] primary dev: ETH0 I1121 17:06:56.443182 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:06:56.443196 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:06:57.450587 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:06:57.450742 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:06:57.450790 3798194 disk_worker.go:723] disk space is not compliant W1121 17:06:57.450794 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:06:57.458940 3798194 disk_worker.go:509] system disk:vda1 I1121 17:06:57.458981 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:06:58.306584 3798194 disk_info.go:125] begin check local disk info of client I1121 17:06:58.310081 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:06:58.310090 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003802c0 0xc000380300] I1121 17:06:58.446062 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:06:59.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:06:59.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:06:59.450503 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:06:59.471021 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:07:06.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:07:06.413262 3798194 memory.go:184] no items to output this cycle I1121 17:07:06.413414 3798194 cpu.go:275] no items to output this cycle E1121 17:07:16.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:07:16.413250 3798194 memory.go:184] no items to output this cycle I1121 17:07:16.413400 3798194 cpu.go:275] no items to output this cycle E1121 17:07:26.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:07:26.412895 3798194 memory.go:191] Add success. I1121 17:07:26.416857 3798194 cpu.go:282] Add success. I1121 17:07:26.441463 3798194 net.go:642] Add success. I1121 17:07:26.442805 3798194 net.go:778] primary dev: ETH0 I1121 17:07:26.442820 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:07:26.442832 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:07:29.450428 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:07:29.450488 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:07:29.450513 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:07:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:07:36.412257 3798194 memory.go:184] no items to output this cycle I1121 17:07:36.412397 3798194 cpu.go:275] no items to output this cycle E1121 17:07:46.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:07:46.413261 3798194 memory.go:184] no items to output this cycle I1121 17:07:46.413405 3798194 cpu.go:275] no items to output this cycle E1121 17:07:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:07:56.413268 3798194 memory.go:191] Add success. W1121 17:07:56.413302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:07:56.413313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:07:56.413317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:07:56.417983 3798194 cpu.go:282] Add success. I1121 17:07:56.441374 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:07:56.441884 3798194 net.go:642] Add success. I1121 17:07:56.443477 3798194 net.go:778] primary dev: ETH0 I1121 17:07:56.443493 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:07:56.443506 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:07:57.445532 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:07:57.445806 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:07:57.445818 3798194 disk_worker.go:723] disk space is not compliant W1121 17:07:57.445822 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:07:57.452472 3798194 disk_worker.go:509] system disk:vda1 I1121 17:07:57.452513 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:07:58.313150 3798194 disk_info.go:125] begin check local disk info of client I1121 17:07:58.316739 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:07:58.316748 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000438500 0xc000438540] I1121 17:07:58.446444 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:07:59.454510 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:07:59.454583 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:07:59.454608 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:07:59.471099 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:08:06.411309 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:08:06.411333 3798194 memory.go:184] no items to output this cycle I1121 17:08:06.411482 3798194 cpu.go:275] no items to output this cycle E1121 17:08:16.412291 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:08:16.412313 3798194 memory.go:184] no items to output this cycle I1121 17:08:16.412462 3798194 cpu.go:275] no items to output this cycle E1121 17:08:26.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:08:26.413263 3798194 memory.go:191] Add success. I1121 17:08:26.417318 3798194 cpu.go:282] Add success. I1121 17:08:26.441655 3798194 net.go:642] Add success. I1121 17:08:26.443363 3798194 net.go:778] primary dev: ETH0 I1121 17:08:26.443377 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:08:26.443392 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:08:29.450131 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:08:29.450207 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:08:29.450234 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:08:36.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:08:36.413251 3798194 memory.go:184] no items to output this cycle I1121 17:08:36.413398 3798194 cpu.go:275] no items to output this cycle E1121 17:08:46.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:08:46.411940 3798194 memory.go:184] no items to output this cycle I1121 17:08:46.412084 3798194 cpu.go:275] no items to output this cycle E1121 17:08:56.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:08:56.413261 3798194 memory.go:191] Add success. W1121 17:08:56.413293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:08:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:08:56.413308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:08:56.417416 3798194 cpu.go:282] Add success. I1121 17:08:56.441801 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:08:56.442301 3798194 net.go:642] Add success. I1121 17:08:56.443842 3798194 net.go:778] primary dev: ETH0 I1121 17:08:56.443856 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:08:56.443868 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:08:57.450581 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:08:57.450753 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:08:57.450808 3798194 disk_worker.go:723] disk space is not compliant W1121 17:08:57.450812 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:08:57.459761 3798194 disk_worker.go:509] system disk:vda1 I1121 17:08:57.459803 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:08:58.316851 3798194 disk_info.go:125] begin check local disk info of client I1121 17:08:58.320470 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:08:58.320479 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a5640 0xc0004a5680] I1121 17:08:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:08:59.450101 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:08:59.450185 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:08:59.450210 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:08:59.471714 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:09:06.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:09:06.413259 3798194 memory.go:184] no items to output this cycle I1121 17:09:06.413405 3798194 cpu.go:275] no items to output this cycle E1121 17:09:16.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:09:16.412251 3798194 memory.go:184] no items to output this cycle I1121 17:09:16.412395 3798194 cpu.go:275] no items to output this cycle I1121 17:09:17.519617 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:09:17.519625 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:09:26.411354 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:09:26.411388 3798194 memory.go:191] Add success. I1121 17:09:26.411729 3798194 cpu.go:282] Add success. I1121 17:09:26.441438 3798194 net.go:642] Add success. I1121 17:09:26.445500 3798194 net.go:778] primary dev: ETH0 I1121 17:09:26.445515 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:09:26.445529 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:09:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:09:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:09:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:09:36.413165 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:09:36.413191 3798194 memory.go:184] no items to output this cycle I1121 17:09:36.413337 3798194 cpu.go:275] no items to output this cycle E1121 17:09:46.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:09:46.413257 3798194 memory.go:184] no items to output this cycle I1121 17:09:46.413401 3798194 cpu.go:275] no items to output this cycle E1121 17:09:56.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:09:56.413273 3798194 memory.go:191] Add success. W1121 17:09:56.413301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:09:56.413313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:09:56.413316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:09:56.417397 3798194 cpu.go:282] Add success. I1121 17:09:56.441931 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:09:56.442508 3798194 net.go:642] Add success. I1121 17:09:56.444039 3798194 net.go:778] primary dev: ETH0 I1121 17:09:56.444053 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:09:56.444065 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:09:57.450755 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:09:57.450964 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:09:57.450975 3798194 disk_worker.go:723] disk space is not compliant W1121 17:09:57.450978 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:09:57.460812 3798194 disk_worker.go:509] system disk:vda1 I1121 17:09:57.460873 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:09:58.323144 3798194 disk_info.go:125] begin check local disk info of client I1121 17:09:58.326666 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:09:58.326675 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d2a80 0xc0003d2ac0] I1121 17:09:58.446391 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:09:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:09:59.450121 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:09:59.450148 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:09:59.471684 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:10:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:10:06.411977 3798194 memory.go:184] no items to output this cycle I1121 17:10:06.412132 3798194 cpu.go:275] no items to output this cycle E1121 17:10:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:10:16.411975 3798194 memory.go:184] no items to output this cycle I1121 17:10:16.412149 3798194 cpu.go:275] no items to output this cycle E1121 17:10:26.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:10:26.411945 3798194 memory.go:191] Add success. I1121 17:10:26.412403 3798194 cpu.go:282] Add success. I1121 17:10:26.441306 3798194 net.go:642] Add success. I1121 17:10:26.445142 3798194 net.go:778] primary dev: ETH0 I1121 17:10:26.445157 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:10:26.445172 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:10:29.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:10:29.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:10:29.450514 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:10:36.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:10:36.412254 3798194 memory.go:184] no items to output this cycle I1121 17:10:36.412405 3798194 cpu.go:275] no items to output this cycle E1121 17:10:46.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:10:46.412887 3798194 memory.go:184] no items to output this cycle I1121 17:10:46.413036 3798194 cpu.go:275] no items to output this cycle E1121 17:10:56.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:10:56.413280 3798194 memory.go:191] Add success. W1121 17:10:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:10:56.413316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:10:56.413319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:10:56.417626 3798194 cpu.go:282] Add success. I1121 17:10:56.443688 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:10:56.444182 3798194 net.go:642] Add success. I1121 17:10:56.445607 3798194 net.go:778] primary dev: ETH0 I1121 17:10:56.445622 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:10:56.445636 3798194 net.go:659] Add eni metrics success, count: 7 E1121 17:10:57.450411 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 17:10:57.450424 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:10:57.450580 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:10:57.450647 3798194 disk_worker.go:723] disk space is not compliant W1121 17:10:57.450650 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:10:57.458817 3798194 disk_worker.go:509] system disk:vda1 I1121 17:10:57.458857 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:10:58.330142 3798194 disk_info.go:125] begin check local disk info of client I1121 17:10:58.333696 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:10:58.333704 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003127c0 0xc000312800] E1121 17:10:58.448186 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 17:10:58.448201 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 17:10:59.448552 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 17:10:59.449628 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:10:59.449681 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:10:59.449700 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:10:59.471187 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:11:06.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:11:06.411978 3798194 memory.go:184] no items to output this cycle I1121 17:11:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 17:11:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:11:16.411977 3798194 memory.go:184] no items to output this cycle I1121 17:11:16.412134 3798194 cpu.go:275] no items to output this cycle E1121 17:11:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:11:26.411954 3798194 memory.go:191] Add success. I1121 17:11:26.412307 3798194 cpu.go:282] Add success. I1121 17:11:26.441944 3798194 net.go:642] Add success. I1121 17:11:26.446017 3798194 net.go:778] primary dev: ETH0 I1121 17:11:26.446032 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:11:26.446045 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:11:29.449890 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:11:29.449972 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:11:29.449998 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:11:36.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:11:36.413254 3798194 memory.go:184] no items to output this cycle I1121 17:11:36.413410 3798194 cpu.go:275] no items to output this cycle E1121 17:11:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:11:46.411982 3798194 memory.go:184] no items to output this cycle I1121 17:11:46.412142 3798194 cpu.go:275] no items to output this cycle E1121 17:11:56.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:11:56.413269 3798194 memory.go:191] Add success. W1121 17:11:56.413295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:11:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:11:56.413310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:11:56.417658 3798194 cpu.go:282] Add success. I1121 17:11:56.441365 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:11:56.441835 3798194 net.go:642] Add success. I1121 17:11:56.443401 3798194 net.go:778] primary dev: ETH0 I1121 17:11:56.443416 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:11:56.443428 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:11:57.445321 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:11:57.445582 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:11:57.445594 3798194 disk_worker.go:723] disk space is not compliant W1121 17:11:57.445597 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:11:57.448094 3798194 disk_worker.go:509] system disk:vda1 I1121 17:11:57.448147 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:11:58.337143 3798194 disk_info.go:125] begin check local disk info of client I1121 17:11:58.340716 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:11:58.340724 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e100 0xc00030e140] I1121 17:11:58.446348 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:11:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:11:59.450133 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:11:59.450161 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:11:59.471645 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:12:06.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:12:06.412884 3798194 memory.go:184] no items to output this cycle I1121 17:12:06.413027 3798194 cpu.go:275] no items to output this cycle E1121 17:12:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:12:16.411978 3798194 memory.go:184] no items to output this cycle I1121 17:12:16.412134 3798194 cpu.go:275] no items to output this cycle I1121 17:12:17.520442 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:12:17.520451 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:12:26.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:12:26.412270 3798194 memory.go:191] Add success. I1121 17:12:26.416886 3798194 cpu.go:282] Add success. I1121 17:12:26.441408 3798194 net.go:642] Add success. I1121 17:12:26.443141 3798194 net.go:778] primary dev: ETH0 I1121 17:12:26.443156 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:12:26.443169 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:12:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:12:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:12:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:12:36.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:12:36.411987 3798194 memory.go:184] no items to output this cycle I1121 17:12:36.412149 3798194 cpu.go:275] no items to output this cycle E1121 17:12:46.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:12:46.412880 3798194 memory.go:184] no items to output this cycle I1121 17:12:46.413026 3798194 cpu.go:275] no items to output this cycle E1121 17:12:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:12:56.411948 3798194 memory.go:191] Add success. W1121 17:12:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:12:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:12:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:12:56.412325 3798194 cpu.go:282] Add success. I1121 17:12:56.441761 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:12:56.442264 3798194 net.go:642] Add success. I1121 17:12:56.446204 3798194 net.go:778] primary dev: ETH0 I1121 17:12:56.446219 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:12:56.446232 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:12:57.450698 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:12:57.450862 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:12:57.450917 3798194 disk_worker.go:723] disk space is not compliant W1121 17:12:57.450921 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:12:57.459162 3798194 disk_worker.go:509] system disk:vda1 I1121 17:12:57.459206 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:12:58.340819 3798194 disk_info.go:125] begin check local disk info of client I1121 17:12:58.344350 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:12:58.344359 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000460d80 0xc000460dc0] I1121 17:12:58.445982 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:12:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:12:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:12:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:12:59.471659 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:13:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:13:06.412259 3798194 memory.go:184] no items to output this cycle I1121 17:13:06.412400 3798194 cpu.go:275] no items to output this cycle E1121 17:13:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:13:16.411989 3798194 memory.go:184] no items to output this cycle I1121 17:13:16.412147 3798194 cpu.go:275] no items to output this cycle E1121 17:13:26.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:13:26.411953 3798194 memory.go:191] Add success. I1121 17:13:26.412327 3798194 cpu.go:282] Add success. I1121 17:13:26.441978 3798194 net.go:642] Add success. I1121 17:13:26.446458 3798194 net.go:778] primary dev: ETH0 I1121 17:13:26.446474 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:13:26.446488 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:13:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:13:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:13:29.450502 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:13:36.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:13:36.412266 3798194 memory.go:184] no items to output this cycle I1121 17:13:36.412414 3798194 cpu.go:275] no items to output this cycle E1121 17:13:46.412955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:13:46.412982 3798194 memory.go:184] no items to output this cycle I1121 17:13:46.413189 3798194 cpu.go:275] no items to output this cycle E1121 17:13:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:13:56.413270 3798194 memory.go:191] Add success. W1121 17:13:56.413299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:13:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:13:56.413314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:13:56.417664 3798194 cpu.go:282] Add success. I1121 17:13:56.441285 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:13:56.441756 3798194 net.go:642] Add success. I1121 17:13:56.443248 3798194 net.go:778] primary dev: ETH0 I1121 17:13:56.443263 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:13:56.443276 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:13:57.450365 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:13:57.450567 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:13:57.450578 3798194 disk_worker.go:723] disk space is not compliant W1121 17:13:57.450581 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:13:57.458680 3798194 disk_worker.go:509] system disk:vda1 I1121 17:13:57.458723 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:13:58.344459 3798194 disk_info.go:125] begin check local disk info of client I1121 17:13:58.347987 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:13:58.347995 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a8240 0xc0003a8280] I1121 17:13:58.446441 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:13:59.450536 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:13:59.450619 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:13:59.450647 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:13:59.471181 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:14:06.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:14:06.412252 3798194 memory.go:184] no items to output this cycle I1121 17:14:06.412404 3798194 cpu.go:275] no items to output this cycle E1121 17:14:16.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:14:16.411993 3798194 memory.go:184] no items to output this cycle I1121 17:14:16.412153 3798194 cpu.go:275] no items to output this cycle E1121 17:14:26.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:14:26.412267 3798194 memory.go:191] Add success. I1121 17:14:26.416443 3798194 cpu.go:282] Add success. I1121 17:14:26.441295 3798194 net.go:642] Add success. I1121 17:14:26.442763 3798194 net.go:778] primary dev: ETH0 I1121 17:14:26.442779 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:14:26.442796 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:14:29.450384 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:14:29.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:14:29.450489 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:14:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:14:36.411978 3798194 memory.go:184] no items to output this cycle I1121 17:14:36.412132 3798194 cpu.go:275] no items to output this cycle E1121 17:14:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:14:46.412256 3798194 memory.go:184] no items to output this cycle I1121 17:14:46.412589 3798194 cpu.go:275] no items to output this cycle E1121 17:14:56.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:14:56.412893 3798194 memory.go:191] Add success. W1121 17:14:56.412920 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:14:56.412931 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:14:56.412935 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:14:56.417312 3798194 cpu.go:282] Add success. I1121 17:14:56.441796 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:14:56.442301 3798194 net.go:642] Add success. I1121 17:14:56.443808 3798194 net.go:778] primary dev: ETH0 I1121 17:14:56.443822 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:14:56.443867 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:14:57.450865 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:14:57.451151 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:14:57.451166 3798194 disk_worker.go:723] disk space is not compliant W1121 17:14:57.451170 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:14:57.459290 3798194 disk_worker.go:509] system disk:vda1 I1121 17:14:57.459342 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:14:58.351566 3798194 disk_info.go:125] begin check local disk info of client I1121 17:14:58.355049 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:14:58.355058 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004483c0 0xc000448400] I1121 17:14:58.446674 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:14:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:14:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:14:59.450146 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:14:59.471683 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:15:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:15:06.412259 3798194 memory.go:184] no items to output this cycle I1121 17:15:06.412407 3798194 cpu.go:275] no items to output this cycle E1121 17:15:16.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:15:16.411972 3798194 memory.go:184] no items to output this cycle I1121 17:15:16.412130 3798194 cpu.go:275] no items to output this cycle I1121 17:15:17.521440 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:15:17.521449 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:15:26.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:15:26.412894 3798194 memory.go:191] Add success. I1121 17:15:26.417142 3798194 cpu.go:282] Add success. I1121 17:15:26.441685 3798194 net.go:642] Add success. I1121 17:15:26.443144 3798194 net.go:778] primary dev: ETH0 I1121 17:15:26.443157 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:15:26.443170 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:15:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:15:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:15:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:15:36.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:15:36.412879 3798194 memory.go:184] no items to output this cycle I1121 17:15:36.413031 3798194 cpu.go:275] no items to output this cycle E1121 17:15:46.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:15:46.412257 3798194 memory.go:184] no items to output this cycle I1121 17:15:46.412402 3798194 cpu.go:275] no items to output this cycle E1121 17:15:56.411933 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:15:56.411968 3798194 memory.go:191] Add success. W1121 17:15:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:15:56.412005 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:15:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:15:56.412352 3798194 cpu.go:282] Add success. I1121 17:15:56.441570 3798194 event_worker.go:152] Polling the log file for events... I1121 17:15:56.444439 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:15:56.444941 3798194 net.go:642] Add success. I1121 17:15:56.448875 3798194 net.go:778] primary dev: ETH0 I1121 17:15:56.448890 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:15:56.448903 3798194 net.go:659] Add eni metrics success, count: 7 E1121 17:15:57.450288 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 17:15:57.450302 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:15:57.450493 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:15:57.450502 3798194 disk_worker.go:723] disk space is not compliant W1121 17:15:57.450505 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:15:57.459317 3798194 disk_worker.go:509] system disk:vda1 I1121 17:15:57.459359 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:15:58.355173 3798194 disk_info.go:125] begin check local disk info of client I1121 17:15:58.359956 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:15:58.359967 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000457400 0xc000457440] E1121 17:15:58.448267 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 17:15:58.448282 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 17:15:59.448616 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 17:15:59.449692 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:15:59.449740 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:15:59.449759 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:15:59.471185 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:16:06.411975 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:16:06.411999 3798194 memory.go:184] no items to output this cycle I1121 17:16:06.412163 3798194 cpu.go:275] no items to output this cycle E1121 17:16:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:16:16.411986 3798194 memory.go:184] no items to output this cycle I1121 17:16:16.412143 3798194 cpu.go:275] no items to output this cycle E1121 17:16:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:16:26.411956 3798194 memory.go:191] Add success. I1121 17:16:26.412309 3798194 cpu.go:282] Add success. I1121 17:16:26.441945 3798194 net.go:642] Add success. I1121 17:16:26.446133 3798194 net.go:778] primary dev: ETH0 I1121 17:16:26.446149 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:16:26.446162 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:16:29.450038 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:16:29.450138 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:16:29.450165 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:16:36.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:16:36.412269 3798194 memory.go:184] no items to output this cycle I1121 17:16:36.412412 3798194 cpu.go:275] no items to output this cycle E1121 17:16:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:16:46.411980 3798194 memory.go:184] no items to output this cycle I1121 17:16:46.412132 3798194 cpu.go:275] no items to output this cycle E1121 17:16:56.411234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:16:56.411267 3798194 memory.go:191] Add success. W1121 17:16:56.411293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:16:56.411305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:16:56.411309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:16:56.415646 3798194 cpu.go:282] Add success. I1121 17:16:56.440946 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:16:56.441541 3798194 net.go:642] Add success. I1121 17:16:56.443098 3798194 net.go:778] primary dev: ETH0 I1121 17:16:56.443124 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:16:56.443138 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:16:57.450252 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:16:57.450467 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:16:57.450477 3798194 disk_worker.go:723] disk space is not compliant W1121 17:16:57.450480 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:16:57.458429 3798194 disk_worker.go:509] system disk:vda1 I1121 17:16:57.458483 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:16:58.360076 3798194 disk_info.go:125] begin check local disk info of client I1121 17:16:58.363648 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:16:58.363657 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f0b80 0xc0003f0bc0] I1121 17:16:58.446157 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:16:59.450029 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:16:59.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:16:59.450147 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:16:59.471713 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:17:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:17:06.411982 3798194 memory.go:184] no items to output this cycle I1121 17:17:06.412132 3798194 cpu.go:275] no items to output this cycle E1121 17:17:16.412362 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:17:16.412388 3798194 memory.go:184] no items to output this cycle I1121 17:17:16.412535 3798194 cpu.go:275] no items to output this cycle E1121 17:17:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:17:26.412049 3798194 memory.go:191] Add success. I1121 17:17:26.412657 3798194 cpu.go:282] Add success. I1121 17:17:26.441304 3798194 net.go:642] Add success. I1121 17:17:26.445492 3798194 net.go:778] primary dev: ETH0 I1121 17:17:26.445506 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:17:26.445519 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:17:29.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:17:29.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:17:29.450153 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:17:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:17:36.411975 3798194 memory.go:184] no items to output this cycle I1121 17:17:36.412129 3798194 cpu.go:275] no items to output this cycle E1121 17:17:46.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:17:46.411987 3798194 memory.go:184] no items to output this cycle I1121 17:17:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 17:17:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:17:56.411945 3798194 memory.go:191] Add success. W1121 17:17:56.411974 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:17:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:17:56.411990 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:17:56.412355 3798194 cpu.go:282] Add success. I1121 17:17:56.441947 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:17:56.442686 3798194 net.go:642] Add success. I1121 17:17:56.446892 3798194 net.go:778] primary dev: ETH0 I1121 17:17:56.446906 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:17:56.446920 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:17:57.450343 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:17:57.450545 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:17:57.450556 3798194 disk_worker.go:723] disk space is not compliant W1121 17:17:57.450559 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:17:57.458772 3798194 disk_worker.go:509] system disk:vda1 I1121 17:17:57.458814 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:17:58.368152 3798194 disk_info.go:125] begin check local disk info of client I1121 17:17:58.371642 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:17:58.371651 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046d000 0xc00046d040] I1121 17:17:58.446128 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:17:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:17:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:17:59.450148 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:17:59.471671 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:18:06.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:18:06.411986 3798194 memory.go:184] no items to output this cycle I1121 17:18:06.412137 3798194 cpu.go:275] no items to output this cycle E1121 17:18:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:18:16.411974 3798194 memory.go:184] no items to output this cycle I1121 17:18:16.412136 3798194 cpu.go:275] no items to output this cycle I1121 17:18:17.522457 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:18:17.522466 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:18:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:18:26.413266 3798194 memory.go:191] Add success. I1121 17:18:26.417440 3798194 cpu.go:282] Add success. I1121 17:18:26.442122 3798194 net.go:642] Add success. I1121 17:18:26.443642 3798194 net.go:778] primary dev: ETH0 I1121 17:18:26.443656 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:18:26.443670 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:18:29.450386 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:18:29.450466 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:18:29.450493 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:18:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:18:36.411980 3798194 memory.go:184] no items to output this cycle I1121 17:18:36.412130 3798194 cpu.go:275] no items to output this cycle E1121 17:18:46.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:18:46.411990 3798194 memory.go:184] no items to output this cycle I1121 17:18:46.412143 3798194 cpu.go:275] no items to output this cycle E1121 17:18:56.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:18:56.412282 3798194 memory.go:191] Add success. W1121 17:18:56.412310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:18:56.412321 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:18:56.412325 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:18:56.416345 3798194 cpu.go:282] Add success. I1121 17:18:56.442686 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:18:56.443186 3798194 net.go:642] Add success. I1121 17:18:56.444886 3798194 net.go:778] primary dev: ETH0 I1121 17:18:56.444902 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:18:56.444914 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:18:57.445273 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:18:57.445539 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:18:57.445551 3798194 disk_worker.go:723] disk space is not compliant W1121 17:18:57.445554 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:18:57.448009 3798194 disk_worker.go:509] system disk:vda1 I1121 17:18:57.448047 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:18:58.376145 3798194 disk_info.go:125] begin check local disk info of client I1121 17:18:58.379710 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:18:58.379725 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0140 0xc0002e0180] I1121 17:18:58.446202 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:18:59.450502 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:18:59.450588 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:18:59.450616 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:18:59.471121 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:19:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:19:06.411976 3798194 memory.go:184] no items to output this cycle I1121 17:19:06.412130 3798194 cpu.go:275] no items to output this cycle E1121 17:19:16.412865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:19:16.412887 3798194 memory.go:184] no items to output this cycle I1121 17:19:16.413038 3798194 cpu.go:275] no items to output this cycle E1121 17:19:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:19:26.411956 3798194 memory.go:191] Add success. I1121 17:19:26.412306 3798194 cpu.go:282] Add success. I1121 17:19:26.441937 3798194 net.go:642] Add success. I1121 17:19:26.445936 3798194 net.go:778] primary dev: ETH0 I1121 17:19:26.445953 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:19:26.445965 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:19:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:19:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:19:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:19:36.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:19:36.412255 3798194 memory.go:184] no items to output this cycle I1121 17:19:36.412394 3798194 cpu.go:275] no items to output this cycle E1121 17:19:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:19:46.411982 3798194 memory.go:184] no items to output this cycle I1121 17:19:46.412142 3798194 cpu.go:275] no items to output this cycle E1121 17:19:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:19:56.411954 3798194 memory.go:191] Add success. W1121 17:19:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:19:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:19:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:19:56.412360 3798194 cpu.go:282] Add success. I1121 17:19:56.441819 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:19:56.442411 3798194 net.go:642] Add success. I1121 17:19:56.446350 3798194 net.go:778] primary dev: ETH0 I1121 17:19:56.446366 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:19:56.446379 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:19:57.450299 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:19:57.450501 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:19:57.450511 3798194 disk_worker.go:723] disk space is not compliant W1121 17:19:57.450514 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:19:57.459251 3798194 disk_worker.go:509] system disk:vda1 I1121 17:19:57.459289 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:19:58.379830 3798194 disk_info.go:125] begin check local disk info of client I1121 17:19:58.383412 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:19:58.383422 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00044f740 0xc00044f780] I1121 17:19:58.445876 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:19:59.450201 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:19:59.450287 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:19:59.450312 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:19:59.470842 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:20:06.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:20:06.412265 3798194 memory.go:184] no items to output this cycle I1121 17:20:06.412413 3798194 cpu.go:275] no items to output this cycle E1121 17:20:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:20:16.413262 3798194 memory.go:184] no items to output this cycle I1121 17:20:16.413410 3798194 cpu.go:275] no items to output this cycle E1121 17:20:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:20:26.411956 3798194 memory.go:191] Add success. I1121 17:20:26.412320 3798194 cpu.go:282] Add success. I1121 17:20:26.442020 3798194 net.go:642] Add success. I1121 17:20:26.446229 3798194 net.go:778] primary dev: ETH0 I1121 17:20:26.446246 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:20:26.446260 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:20:29.450029 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:20:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:20:29.450145 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:20:36.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:20:36.412267 3798194 memory.go:184] no items to output this cycle I1121 17:20:36.412420 3798194 cpu.go:275] no items to output this cycle E1121 17:20:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:20:46.411983 3798194 memory.go:184] no items to output this cycle I1121 17:20:46.412144 3798194 cpu.go:275] no items to output this cycle E1121 17:20:56.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:20:56.413267 3798194 memory.go:191] Add success. W1121 17:20:56.413300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:20:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:20:56.413314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:20:56.417585 3798194 cpu.go:282] Add success. I1121 17:20:56.443750 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:20:56.444255 3798194 net.go:642] Add success. I1121 17:20:56.445929 3798194 net.go:778] primary dev: ETH0 I1121 17:20:56.445944 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:20:56.445956 3798194 net.go:659] Add eni metrics success, count: 7 E1121 17:20:57.450875 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 17:20:57.450888 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:20:57.451083 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:20:57.451092 3798194 disk_worker.go:723] disk space is not compliant W1121 17:20:57.451095 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:20:57.459312 3798194 disk_worker.go:509] system disk:vda1 I1121 17:20:57.459354 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:20:58.387141 3798194 disk_info.go:125] begin check local disk info of client I1121 17:20:58.390701 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:20:58.390711 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ca2c0 0xc0003ca300] E1121 17:20:58.447819 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 17:20:58.447833 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 17:20:59.448521 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 17:20:59.450650 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:20:59.450700 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:20:59.450721 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:20:59.471237 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:21:06.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:21:06.413263 3798194 memory.go:184] no items to output this cycle I1121 17:21:06.413408 3798194 cpu.go:275] no items to output this cycle E1121 17:21:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:21:16.411978 3798194 memory.go:184] no items to output this cycle I1121 17:21:16.412142 3798194 cpu.go:275] no items to output this cycle I1121 17:21:17.523302 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:21:17.523311 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:21:26.411226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:21:26.411257 3798194 memory.go:191] Add success. I1121 17:21:26.411725 3798194 cpu.go:282] Add success. I1121 17:21:26.441260 3798194 net.go:642] Add success. I1121 17:21:26.445307 3798194 net.go:778] primary dev: ETH0 I1121 17:21:26.445323 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:21:26.445336 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:21:29.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:21:29.450137 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:21:29.450169 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:21:36.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:21:36.411984 3798194 memory.go:184] no items to output this cycle I1121 17:21:36.412148 3798194 cpu.go:275] no items to output this cycle E1121 17:21:46.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:21:46.413265 3798194 memory.go:184] no items to output this cycle I1121 17:21:46.413418 3798194 cpu.go:275] no items to output this cycle E1121 17:21:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:21:56.411952 3798194 memory.go:191] Add success. W1121 17:21:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:21:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:21:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:21:56.412354 3798194 cpu.go:282] Add success. I1121 17:21:56.441934 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:21:56.442536 3798194 net.go:642] Add success. I1121 17:21:56.446667 3798194 net.go:778] primary dev: ETH0 I1121 17:21:56.446684 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:21:56.446695 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:21:57.450691 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:21:57.450896 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:21:57.450909 3798194 disk_worker.go:723] disk space is not compliant W1121 17:21:57.450912 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:21:57.458996 3798194 disk_worker.go:509] system disk:vda1 I1121 17:21:57.459037 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:21:58.394143 3798194 disk_info.go:125] begin check local disk info of client I1121 17:21:58.397743 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:21:58.397752 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00027e740 0xc00027e780] I1121 17:21:58.446128 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:21:59.450193 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:21:59.450274 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:21:59.450300 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:21:59.470845 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:22:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:22:06.411977 3798194 memory.go:184] no items to output this cycle I1121 17:22:06.412144 3798194 cpu.go:275] no items to output this cycle E1121 17:22:16.411972 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:22:16.411997 3798194 memory.go:184] no items to output this cycle I1121 17:22:16.412165 3798194 cpu.go:275] no items to output this cycle E1121 17:22:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:22:26.411956 3798194 memory.go:191] Add success. I1121 17:22:26.412331 3798194 cpu.go:282] Add success. I1121 17:22:26.441976 3798194 net.go:642] Add success. I1121 17:22:26.445985 3798194 net.go:778] primary dev: ETH0 I1121 17:22:26.446001 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:22:26.446016 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:22:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:22:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:22:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:22:36.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:22:36.412249 3798194 memory.go:184] no items to output this cycle I1121 17:22:36.412403 3798194 cpu.go:275] no items to output this cycle E1121 17:22:46.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:22:46.413260 3798194 memory.go:184] no items to output this cycle I1121 17:22:46.413407 3798194 cpu.go:275] no items to output this cycle E1121 17:22:56.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:22:56.412271 3798194 memory.go:191] Add success. W1121 17:22:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:22:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:22:56.412317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:22:56.416546 3798194 cpu.go:282] Add success. I1121 17:22:56.440962 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:22:56.441570 3798194 net.go:642] Add success. I1121 17:22:56.443129 3798194 net.go:778] primary dev: ETH0 I1121 17:22:56.443143 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:22:56.443155 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:22:57.456396 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:22:57.456565 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:22:57.456619 3798194 disk_worker.go:723] disk space is not compliant W1121 17:22:57.456622 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:22:57.464736 3798194 disk_worker.go:509] system disk:vda1 I1121 17:22:57.464777 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:22:58.402149 3798194 disk_info.go:125] begin check local disk info of client I1121 17:22:58.405762 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:22:58.405771 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071f00 0xc000071f40] I1121 17:22:58.446132 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:22:59.450497 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:22:59.450586 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:22:59.450624 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:22:59.471460 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:23:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:23:06.411979 3798194 memory.go:184] no items to output this cycle I1121 17:23:06.412134 3798194 cpu.go:275] no items to output this cycle E1121 17:23:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:23:16.411983 3798194 memory.go:184] no items to output this cycle I1121 17:23:16.412144 3798194 cpu.go:275] no items to output this cycle E1121 17:23:26.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:23:26.411944 3798194 memory.go:191] Add success. I1121 17:23:26.412302 3798194 cpu.go:282] Add success. I1121 17:23:26.442058 3798194 net.go:642] Add success. I1121 17:23:26.446124 3798194 net.go:778] primary dev: ETH0 I1121 17:23:26.446141 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:23:26.446154 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:23:29.449898 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:23:29.449987 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:23:29.450015 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:23:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:23:36.411977 3798194 memory.go:184] no items to output this cycle I1121 17:23:36.412130 3798194 cpu.go:275] no items to output this cycle E1121 17:23:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:23:46.411975 3798194 memory.go:184] no items to output this cycle I1121 17:23:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 17:23:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:23:56.413268 3798194 memory.go:191] Add success. W1121 17:23:56.413295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:23:56.413307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:23:56.413310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:23:56.417614 3798194 cpu.go:282] Add success. I1121 17:23:56.440967 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:23:56.441557 3798194 net.go:642] Add success. I1121 17:23:56.443337 3798194 net.go:778] primary dev: ETH0 I1121 17:23:56.443353 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:23:56.443367 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:23:57.450388 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:23:57.450602 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:23:57.450613 3798194 disk_worker.go:723] disk space is not compliant W1121 17:23:57.450616 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:23:57.458660 3798194 disk_worker.go:509] system disk:vda1 I1121 17:23:57.458701 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:23:58.410359 3798194 disk_info.go:125] begin check local disk info of client I1121 17:23:58.414006 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:23:58.414015 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040bb00 0xc00040bb40] I1121 17:23:58.446190 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:23:59.450280 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:23:59.450372 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:23:59.450398 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:23:59.470925 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:24:06.412868 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:24:06.412892 3798194 memory.go:184] no items to output this cycle I1121 17:24:06.413262 3798194 cpu.go:275] no items to output this cycle E1121 17:24:16.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:24:16.413252 3798194 memory.go:184] no items to output this cycle I1121 17:24:16.413409 3798194 cpu.go:275] no items to output this cycle I1121 17:24:17.524081 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:24:17.524090 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:24:26.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:24:26.413274 3798194 memory.go:191] Add success. I1121 17:24:26.417730 3798194 cpu.go:282] Add success. I1121 17:24:26.442283 3798194 net.go:642] Add success. I1121 17:24:26.443777 3798194 net.go:778] primary dev: ETH0 I1121 17:24:26.443791 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:24:26.443804 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:24:29.450558 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:24:29.450636 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:24:29.450665 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:24:36.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:24:36.413264 3798194 memory.go:184] no items to output this cycle I1121 17:24:36.413413 3798194 cpu.go:275] no items to output this cycle E1121 17:24:46.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:24:46.412253 3798194 memory.go:184] no items to output this cycle I1121 17:24:46.412399 3798194 cpu.go:275] no items to output this cycle E1121 17:24:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:24:56.411962 3798194 memory.go:191] Add success. W1121 17:24:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:24:56.412010 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:24:56.412013 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:24:56.412358 3798194 cpu.go:282] Add success. I1121 17:24:56.441904 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:24:56.442457 3798194 net.go:642] Add success. I1121 17:24:56.446390 3798194 net.go:778] primary dev: ETH0 I1121 17:24:56.446405 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:24:56.446420 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:24:57.450414 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:24:57.450577 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:24:57.450633 3798194 disk_worker.go:723] disk space is not compliant W1121 17:24:57.450637 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:24:57.458695 3798194 disk_worker.go:509] system disk:vda1 I1121 17:24:57.458737 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:24:58.414210 3798194 disk_info.go:125] begin check local disk info of client I1121 17:24:58.417880 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:24:58.417889 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037ce80 0xc00037cec0] I1121 17:24:58.446123 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:24:59.450219 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:24:59.450319 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:24:59.450347 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:24:59.470853 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:25:06.412764 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:25:06.412786 3798194 memory.go:184] no items to output this cycle I1121 17:25:06.412936 3798194 cpu.go:275] no items to output this cycle E1121 17:25:16.413129 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:25:16.413151 3798194 memory.go:184] no items to output this cycle I1121 17:25:16.413301 3798194 cpu.go:275] no items to output this cycle E1121 17:25:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:25:26.411947 3798194 memory.go:191] Add success. I1121 17:25:26.412315 3798194 cpu.go:282] Add success. I1121 17:25:26.441922 3798194 net.go:642] Add success. I1121 17:25:26.446359 3798194 net.go:778] primary dev: ETH0 I1121 17:25:26.446378 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:25:26.446391 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:25:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:25:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:25:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:25:36.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:25:36.412250 3798194 memory.go:184] no items to output this cycle I1121 17:25:36.412392 3798194 cpu.go:275] no items to output this cycle E1121 17:25:46.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:25:46.412893 3798194 memory.go:184] no items to output this cycle I1121 17:25:46.413056 3798194 cpu.go:275] no items to output this cycle E1121 17:25:56.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:25:56.413270 3798194 memory.go:191] Add success. W1121 17:25:56.413298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:25:56.413310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:25:56.413313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:25:56.417772 3798194 cpu.go:282] Add success. I1121 17:25:56.443586 3798194 event_worker.go:152] Polling the log file for events... I1121 17:25:56.443797 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:25:56.444379 3798194 net.go:642] Add success. I1121 17:25:56.445919 3798194 net.go:778] primary dev: ETH0 I1121 17:25:56.445933 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:25:56.445947 3798194 net.go:659] Add eni metrics success, count: 7 E1121 17:25:57.450654 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 17:25:57.450667 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:25:57.450862 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:25:57.450871 3798194 disk_worker.go:723] disk space is not compliant W1121 17:25:57.450874 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:25:57.458944 3798194 disk_worker.go:509] system disk:vda1 I1121 17:25:57.458987 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:25:58.418287 3798194 disk_info.go:125] begin check local disk info of client I1121 17:25:58.421906 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:25:58.421916 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037d080 0xc00037d0c0] E1121 17:25:58.447650 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 17:25:58.447663 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 17:25:59.448949 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 17:25:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:25:59.450074 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:25:59.450092 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:25:59.471639 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:26:06.413262 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:26:06.413283 3798194 memory.go:184] no items to output this cycle I1121 17:26:06.413429 3798194 cpu.go:275] no items to output this cycle E1121 17:26:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:26:16.413261 3798194 memory.go:184] no items to output this cycle I1121 17:26:16.413411 3798194 cpu.go:275] no items to output this cycle E1121 17:26:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:26:26.413269 3798194 memory.go:191] Add success. I1121 17:26:26.417384 3798194 cpu.go:282] Add success. I1121 17:26:26.441904 3798194 net.go:642] Add success. I1121 17:26:26.443528 3798194 net.go:778] primary dev: ETH0 I1121 17:26:26.443542 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:26:26.443557 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:26:29.449975 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:26:29.450050 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:26:29.450075 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:26:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:26:36.412254 3798194 memory.go:184] no items to output this cycle I1121 17:26:36.412396 3798194 cpu.go:275] no items to output this cycle E1121 17:26:46.413095 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:26:46.413132 3798194 memory.go:184] no items to output this cycle I1121 17:26:46.413282 3798194 cpu.go:275] no items to output this cycle E1121 17:26:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:26:56.411956 3798194 memory.go:191] Add success. W1121 17:26:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:26:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:26:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:26:56.412334 3798194 cpu.go:282] Add success. I1121 17:26:56.441799 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:26:56.442315 3798194 net.go:642] Add success. I1121 17:26:56.446243 3798194 net.go:778] primary dev: ETH0 I1121 17:26:56.446258 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:26:56.446269 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:26:57.450715 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:26:57.450928 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:26:57.450939 3798194 disk_worker.go:723] disk space is not compliant W1121 17:26:57.450942 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:26:57.459842 3798194 disk_worker.go:509] system disk:vda1 I1121 17:26:57.459881 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:26:58.422897 3798194 disk_info.go:125] begin check local disk info of client I1121 17:26:58.426471 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:26:58.426481 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000491640 0xc000491680] I1121 17:26:58.446659 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:26:59.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:26:59.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:26:59.450503 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:26:59.471027 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:27:06.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:27:06.413265 3798194 memory.go:184] no items to output this cycle I1121 17:27:06.413409 3798194 cpu.go:275] no items to output this cycle E1121 17:27:16.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:27:16.412263 3798194 memory.go:184] no items to output this cycle I1121 17:27:16.412408 3798194 cpu.go:275] no items to output this cycle I1121 17:27:17.524250 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:27:17.524259 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:27:26.411214 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:27:26.411249 3798194 memory.go:191] Add success. I1121 17:27:26.411575 3798194 cpu.go:282] Add success. I1121 17:27:26.442245 3798194 net.go:642] Add success. I1121 17:27:26.446135 3798194 net.go:778] primary dev: ETH0 I1121 17:27:26.446149 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:27:26.446166 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:27:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:27:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:27:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:27:36.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:27:36.412252 3798194 memory.go:184] no items to output this cycle I1121 17:27:36.412394 3798194 cpu.go:275] no items to output this cycle E1121 17:27:46.412869 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:27:46.412892 3798194 memory.go:184] no items to output this cycle I1121 17:27:46.413038 3798194 cpu.go:275] no items to output this cycle E1121 17:27:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:27:56.411949 3798194 memory.go:191] Add success. W1121 17:27:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:27:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:27:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:27:56.412360 3798194 cpu.go:282] Add success. I1121 17:27:56.441819 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:27:56.442344 3798194 net.go:642] Add success. I1121 17:27:56.446454 3798194 net.go:778] primary dev: ETH0 I1121 17:27:56.446469 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:27:56.446481 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:27:57.450314 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:27:57.450481 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:27:57.450536 3798194 disk_worker.go:723] disk space is not compliant W1121 17:27:57.450539 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:27:57.458587 3798194 disk_worker.go:509] system disk:vda1 I1121 17:27:57.458630 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:27:58.426904 3798194 disk_info.go:125] begin check local disk info of client I1121 17:27:58.430493 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:27:58.430503 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d6a80 0xc0003d6ac0] I1121 17:27:58.446661 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:27:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:27:59.450119 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:27:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:27:59.471672 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:28:06.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:28:06.411981 3798194 memory.go:184] no items to output this cycle I1121 17:28:06.412133 3798194 cpu.go:275] no items to output this cycle E1121 17:28:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:28:16.411974 3798194 memory.go:184] no items to output this cycle I1121 17:28:16.412147 3798194 cpu.go:275] no items to output this cycle E1121 17:28:26.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:28:26.412261 3798194 memory.go:191] Add success. I1121 17:28:26.416477 3798194 cpu.go:282] Add success. I1121 17:28:26.441295 3798194 net.go:642] Add success. I1121 17:28:26.442926 3798194 net.go:778] primary dev: ETH0 I1121 17:28:26.442941 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:28:26.442956 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:28:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:28:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:28:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:28:36.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:28:36.411982 3798194 memory.go:184] no items to output this cycle I1121 17:28:36.412139 3798194 cpu.go:275] no items to output this cycle E1121 17:28:46.412005 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:28:46.412034 3798194 memory.go:184] no items to output this cycle I1121 17:28:46.412250 3798194 cpu.go:275] no items to output this cycle E1121 17:28:56.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:28:56.413265 3798194 memory.go:191] Add success. W1121 17:28:56.413293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:28:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:28:56.413308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:28:56.417949 3798194 cpu.go:282] Add success. I1121 17:28:56.441305 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:28:56.441794 3798194 net.go:642] Add success. I1121 17:28:56.443570 3798194 net.go:778] primary dev: ETH0 I1121 17:28:56.443586 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:28:56.443599 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:28:57.445281 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:28:57.445491 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:28:57.445555 3798194 disk_worker.go:723] disk space is not compliant W1121 17:28:57.445558 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:28:57.448087 3798194 disk_worker.go:509] system disk:vda1 I1121 17:28:57.448141 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:28:58.431314 3798194 disk_info.go:125] begin check local disk info of client I1121 17:28:58.434947 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:28:58.434964 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00027fc80 0xc00027fcc0] I1121 17:28:58.446048 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:28:59.450581 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:28:59.450645 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:28:59.450671 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:28:59.471195 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:29:06.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:29:06.412878 3798194 memory.go:184] no items to output this cycle I1121 17:29:06.413023 3798194 cpu.go:275] no items to output this cycle E1121 17:29:16.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:29:16.412257 3798194 memory.go:184] no items to output this cycle I1121 17:29:16.412403 3798194 cpu.go:275] no items to output this cycle E1121 17:29:26.411934 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:29:26.411968 3798194 memory.go:191] Add success. I1121 17:29:26.412327 3798194 cpu.go:282] Add success. I1121 17:29:26.441957 3798194 net.go:642] Add success. I1121 17:29:26.445959 3798194 net.go:778] primary dev: ETH0 I1121 17:29:26.445975 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:29:26.445989 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:29:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:29:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:29:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:29:36.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:29:36.412261 3798194 memory.go:184] no items to output this cycle I1121 17:29:36.412417 3798194 cpu.go:275] no items to output this cycle E1121 17:29:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:29:46.411977 3798194 memory.go:184] no items to output this cycle I1121 17:29:46.412140 3798194 cpu.go:275] no items to output this cycle E1121 17:29:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:29:56.413263 3798194 memory.go:191] Add success. W1121 17:29:56.413299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:29:56.413310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:29:56.413313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:29:56.417316 3798194 cpu.go:282] Add success. I1121 17:29:56.441892 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:29:56.442897 3798194 net.go:642] Add success. I1121 17:29:56.444533 3798194 net.go:778] primary dev: ETH0 I1121 17:29:56.444548 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:29:56.444561 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:29:57.450765 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:29:57.450931 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:29:57.450986 3798194 disk_worker.go:723] disk space is not compliant W1121 17:29:57.450990 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:29:57.459220 3798194 disk_worker.go:509] system disk:vda1 I1121 17:29:57.459263 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:29:58.435346 3798194 disk_info.go:125] begin check local disk info of client I1121 17:29:58.438914 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:29:58.438924 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00048c8c0 0xc00048c900] I1121 17:29:58.446058 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:29:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:29:59.450125 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:29:59.450157 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:29:59.471718 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:30:06.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:30:06.411987 3798194 memory.go:184] no items to output this cycle I1121 17:30:06.412144 3798194 cpu.go:275] no items to output this cycle E1121 17:30:16.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:30:16.412261 3798194 memory.go:184] no items to output this cycle I1121 17:30:16.412424 3798194 cpu.go:275] no items to output this cycle I1121 17:30:17.524418 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:30:17.524427 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:30:26.411162 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:30:26.411198 3798194 memory.go:191] Add success. I1121 17:30:26.411551 3798194 cpu.go:282] Add success. I1121 17:30:26.442161 3798194 net.go:642] Add success. I1121 17:30:26.446053 3798194 net.go:778] primary dev: ETH0 I1121 17:30:26.446068 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:30:26.446081 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:30:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:30:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:30:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:30:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:30:36.413253 3798194 memory.go:184] no items to output this cycle I1121 17:30:36.413403 3798194 cpu.go:275] no items to output this cycle E1121 17:30:46.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:30:46.412267 3798194 memory.go:184] no items to output this cycle I1121 17:30:46.412418 3798194 cpu.go:275] no items to output this cycle E1121 17:30:56.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:30:56.412886 3798194 memory.go:191] Add success. W1121 17:30:56.412918 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:30:56.412929 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:30:56.412933 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:30:56.417203 3798194 cpu.go:282] Add success. I1121 17:30:56.444170 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:30:56.444655 3798194 net.go:642] Add success. I1121 17:30:56.446085 3798194 net.go:778] primary dev: ETH0 I1121 17:30:56.446099 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:30:56.446124 3798194 net.go:659] Add eni metrics success, count: 7 E1121 17:30:57.450088 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 17:30:57.450103 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:30:57.450320 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:30:57.450330 3798194 disk_worker.go:723] disk space is not compliant W1121 17:30:57.450333 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:30:57.458320 3798194 disk_worker.go:509] system disk:vda1 I1121 17:30:57.458366 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:30:58.439224 3798194 disk_info.go:125] begin check local disk info of client I1121 17:30:58.442803 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:30:58.442812 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037e840 0xc00037e880] E1121 17:30:58.447298 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 17:30:58.447309 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 17:30:59.448618 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 17:30:59.449694 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:30:59.449745 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:30:59.449764 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:30:59.471260 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:31:06.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:31:06.412881 3798194 memory.go:184] no items to output this cycle I1121 17:31:06.413258 3798194 cpu.go:275] no items to output this cycle E1121 17:31:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:31:16.412258 3798194 memory.go:184] no items to output this cycle I1121 17:31:16.412409 3798194 cpu.go:275] no items to output this cycle E1121 17:31:26.413994 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:31:26.414040 3798194 memory.go:191] Add success. I1121 17:31:26.418302 3798194 cpu.go:282] Add success. I1121 17:31:26.441824 3798194 net.go:642] Add success. I1121 17:31:26.443346 3798194 net.go:778] primary dev: ETH0 I1121 17:31:26.443362 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:31:26.443374 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:31:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:31:29.450126 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:31:29.450158 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:31:36.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:31:36.411972 3798194 memory.go:184] no items to output this cycle I1121 17:31:36.412130 3798194 cpu.go:275] no items to output this cycle E1121 17:31:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:31:46.411985 3798194 memory.go:184] no items to output this cycle I1121 17:31:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 17:31:56.411929 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:31:56.411962 3798194 memory.go:191] Add success. W1121 17:31:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:31:56.412006 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:31:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:31:56.412381 3798194 cpu.go:282] Add success. I1121 17:31:56.441833 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:31:56.442432 3798194 net.go:642] Add success. I1121 17:31:56.446628 3798194 net.go:778] primary dev: ETH0 I1121 17:31:56.446643 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:31:56.446657 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:31:57.450518 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:31:57.450741 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:31:57.450752 3798194 disk_worker.go:723] disk space is not compliant W1121 17:31:57.450755 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:31:57.459174 3798194 disk_worker.go:509] system disk:vda1 I1121 17:31:57.459216 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:31:58.442914 3798194 disk_info.go:125] begin check local disk info of client I1121 17:31:58.446928 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:31:58.446955 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:31:58.446961 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000372ec0 0xc000372f00] I1121 17:31:59.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:31:59.450489 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:31:59.450518 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:31:59.471040 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:32:06.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:32:06.412267 3798194 memory.go:184] no items to output this cycle I1121 17:32:06.412410 3798194 cpu.go:275] no items to output this cycle E1121 17:32:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:32:16.411976 3798194 memory.go:184] no items to output this cycle I1121 17:32:16.412142 3798194 cpu.go:275] no items to output this cycle E1121 17:32:26.412868 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:32:26.412905 3798194 memory.go:191] Add success. I1121 17:32:26.417874 3798194 cpu.go:282] Add success. I1121 17:32:26.441580 3798194 net.go:642] Add success. I1121 17:32:26.443067 3798194 net.go:778] primary dev: ETH0 I1121 17:32:26.443084 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:32:26.443097 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:32:29.450034 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:32:29.450139 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:32:29.450167 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:32:36.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:32:36.412991 3798194 memory.go:184] no items to output this cycle I1121 17:32:36.413292 3798194 cpu.go:275] no items to output this cycle E1121 17:32:46.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:32:46.411987 3798194 memory.go:184] no items to output this cycle I1121 17:32:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 17:32:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:32:56.412270 3798194 memory.go:191] Add success. W1121 17:32:56.412298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:32:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:32:56.412313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:32:56.416448 3798194 cpu.go:282] Add success. I1121 17:32:56.440942 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:32:56.441519 3798194 net.go:642] Add success. I1121 17:32:56.443057 3798194 net.go:778] primary dev: ETH0 I1121 17:32:56.443071 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:32:56.443083 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:32:57.445349 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:32:57.445563 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:32:57.445627 3798194 disk_worker.go:723] disk space is not compliant W1121 17:32:57.445630 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:32:57.448160 3798194 disk_worker.go:509] system disk:vda1 I1121 17:32:57.448203 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:32:58.445990 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:32:58.447040 3798194 disk_info.go:125] begin check local disk info of client I1121 17:32:58.450751 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:32:58.450761 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00043ad40 0xc00043ad80] I1121 17:32:59.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:32:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:32:59.450511 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:32:59.471019 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:33:06.412338 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:33:06.412361 3798194 memory.go:184] no items to output this cycle I1121 17:33:06.412511 3798194 cpu.go:275] no items to output this cycle E1121 17:33:16.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:33:16.411986 3798194 memory.go:184] no items to output this cycle I1121 17:33:16.412145 3798194 cpu.go:275] no items to output this cycle I1121 17:33:17.525477 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:33:17.525485 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:33:26.411277 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:33:26.411310 3798194 memory.go:191] Add success. I1121 17:33:26.411617 3798194 cpu.go:282] Add success. I1121 17:33:26.441295 3798194 net.go:642] Add success. I1121 17:33:26.445423 3798194 net.go:778] primary dev: ETH0 I1121 17:33:26.445439 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:33:26.445454 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:33:29.450045 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:33:29.450136 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:33:29.450162 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:33:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:33:36.413265 3798194 memory.go:184] no items to output this cycle I1121 17:33:36.413409 3798194 cpu.go:275] no items to output this cycle E1121 17:33:46.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:33:46.412255 3798194 memory.go:184] no items to output this cycle I1121 17:33:46.412402 3798194 cpu.go:275] no items to output this cycle E1121 17:33:56.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:33:56.411962 3798194 memory.go:191] Add success. W1121 17:33:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:33:56.412010 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:33:56.412013 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:33:56.412367 3798194 cpu.go:282] Add success. I1121 17:33:56.441820 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:33:56.442421 3798194 net.go:642] Add success. I1121 17:33:56.446619 3798194 net.go:778] primary dev: ETH0 I1121 17:33:56.446637 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:33:56.446652 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:33:57.450386 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:33:57.450617 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:33:57.450629 3798194 disk_worker.go:723] disk space is not compliant W1121 17:33:57.450632 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:33:57.458775 3798194 disk_worker.go:509] system disk:vda1 I1121 17:33:57.458815 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:33:58.446377 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:33:58.451533 3798194 disk_info.go:125] begin check local disk info of client I1121 17:33:58.455269 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:33:58.455278 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000070dc0 0xc000070e00] I1121 17:33:59.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:33:59.450128 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:33:59.450154 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:33:59.471711 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:34:06.412976 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:34:06.412998 3798194 memory.go:184] no items to output this cycle I1121 17:34:06.413165 3798194 cpu.go:275] no items to output this cycle E1121 17:34:16.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:34:16.411986 3798194 memory.go:184] no items to output this cycle I1121 17:34:16.412148 3798194 cpu.go:275] no items to output this cycle E1121 17:34:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:34:26.411961 3798194 memory.go:191] Add success. I1121 17:34:26.412325 3798194 cpu.go:282] Add success. I1121 17:34:26.441976 3798194 net.go:642] Add success. I1121 17:34:26.446311 3798194 net.go:778] primary dev: ETH0 I1121 17:34:26.446328 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:34:26.446341 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:34:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:34:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:34:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:34:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:34:36.412263 3798194 memory.go:184] no items to output this cycle I1121 17:34:36.412405 3798194 cpu.go:275] no items to output this cycle E1121 17:34:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:34:46.411976 3798194 memory.go:184] no items to output this cycle I1121 17:34:46.412130 3798194 cpu.go:275] no items to output this cycle E1121 17:34:56.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:34:56.412894 3798194 memory.go:191] Add success. W1121 17:34:56.412925 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:34:56.412937 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:34:56.412942 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:34:56.417381 3798194 cpu.go:282] Add success. I1121 17:34:56.441513 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:34:56.442004 3798194 net.go:642] Add success. I1121 17:34:56.443471 3798194 net.go:778] primary dev: ETH0 I1121 17:34:56.443488 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:34:56.443501 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:34:57.445249 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:34:57.445463 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:34:57.445532 3798194 disk_worker.go:723] disk space is not compliant W1121 17:34:57.445536 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:34:57.448068 3798194 disk_worker.go:509] system disk:vda1 I1121 17:34:57.448121 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:34:58.445993 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:34:58.456127 3798194 disk_info.go:125] begin check local disk info of client I1121 17:34:58.459725 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:34:58.459734 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d7d00 0xc0003d7d40] I1121 17:34:59.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:34:59.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:34:59.450146 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:34:59.471645 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:35:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:35:06.411984 3798194 memory.go:184] no items to output this cycle I1121 17:35:06.412144 3798194 cpu.go:275] no items to output this cycle E1121 17:35:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:35:16.412073 3798194 memory.go:184] no items to output this cycle I1121 17:35:16.412422 3798194 cpu.go:275] no items to output this cycle E1121 17:35:26.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:35:26.411960 3798194 memory.go:191] Add success. I1121 17:35:26.412339 3798194 cpu.go:282] Add success. I1121 17:35:26.441962 3798194 net.go:642] Add success. I1121 17:35:26.448920 3798194 net.go:778] primary dev: ETH0 I1121 17:35:26.448942 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:35:26.448965 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:35:29.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:35:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:35:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:35:36.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:35:36.412269 3798194 memory.go:184] no items to output this cycle I1121 17:35:36.412422 3798194 cpu.go:275] no items to output this cycle E1121 17:35:46.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:35:46.412263 3798194 memory.go:184] no items to output this cycle I1121 17:35:46.412411 3798194 cpu.go:275] no items to output this cycle E1121 17:35:56.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:35:56.412271 3798194 memory.go:191] Add success. W1121 17:35:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:35:56.412316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:35:56.412319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:35:56.416416 3798194 cpu.go:282] Add success. I1121 17:35:56.441953 3798194 event_worker.go:152] Polling the log file for events... I1121 17:35:56.442366 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:35:56.442833 3798194 net.go:642] Add success. I1121 17:35:56.445947 3798194 net.go:778] primary dev: ETH0 I1121 17:35:56.445962 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:35:56.445974 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:35:56.445989 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795251956, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 17:35:56.451976 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 17:35:57.445380 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 17:35:57.445396 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:35:57.445655 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:35:57.445666 3798194 disk_worker.go:723] disk space is not compliant W1121 17:35:57.445669 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:35:57.448184 3798194 disk_worker.go:509] system disk:vda1 I1121 17:35:57.448224 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 17:35:58.447826 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 17:35:58.447841 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:35:58.459962 3798194 disk_info.go:125] begin check local disk info of client I1121 17:35:58.463196 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:35:58.463204 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000370340 0xc000370380] E1121 17:35:59.448914 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 17:35:59.449989 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:35:59.450045 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:35:59.450064 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:35:59.471570 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:36:06.412479 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:36:06.412502 3798194 memory.go:184] no items to output this cycle I1121 17:36:06.412649 3798194 cpu.go:275] no items to output this cycle E1121 17:36:16.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:36:16.412876 3798194 memory.go:184] no items to output this cycle I1121 17:36:16.413020 3798194 cpu.go:275] no items to output this cycle I1121 17:36:17.526471 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:36:17.526479 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:36:26.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:36:26.412882 3798194 memory.go:191] Add success. I1121 17:36:26.416813 3798194 cpu.go:282] Add success. I1121 17:36:26.441296 3798194 net.go:642] Add success. I1121 17:36:26.442763 3798194 net.go:778] primary dev: ETH0 I1121 17:36:26.442776 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:36:26.442789 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:36:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:36:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:36:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:36:36.411998 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:36:36.412028 3798194 memory.go:184] no items to output this cycle I1121 17:36:36.412525 3798194 cpu.go:275] no items to output this cycle E1121 17:36:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:36:46.411982 3798194 memory.go:184] no items to output this cycle I1121 17:36:46.412143 3798194 cpu.go:275] no items to output this cycle E1121 17:36:56.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:36:56.412274 3798194 memory.go:191] Add success. W1121 17:36:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:36:56.412317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:36:56.412320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:36:56.416640 3798194 cpu.go:282] Add success. I1121 17:36:56.440933 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:36:56.441501 3798194 net.go:642] Add success. I1121 17:36:56.442970 3798194 net.go:778] primary dev: ETH0 I1121 17:36:56.442987 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:36:56.443000 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:36:57.448984 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:36:57.449197 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:36:57.449259 3798194 disk_worker.go:723] disk space is not compliant W1121 17:36:57.449262 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:36:57.451090 3798194 disk_worker.go:509] system disk:vda1 I1121 17:36:57.451151 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:36:58.446508 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:36:58.463682 3798194 disk_info.go:125] begin check local disk info of client I1121 17:36:58.467286 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:36:58.467300 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003705c0 0xc000370600] I1121 17:36:59.450031 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:36:59.450127 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:36:59.450156 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:36:59.471645 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:37:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:37:06.412256 3798194 memory.go:184] no items to output this cycle I1121 17:37:06.412405 3798194 cpu.go:275] no items to output this cycle E1121 17:37:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:37:16.412259 3798194 memory.go:184] no items to output this cycle I1121 17:37:16.412405 3798194 cpu.go:275] no items to output this cycle E1121 17:37:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:37:26.411953 3798194 memory.go:191] Add success. I1121 17:37:26.412301 3798194 cpu.go:282] Add success. I1121 17:37:26.441931 3798194 net.go:642] Add success. I1121 17:37:26.445684 3798194 net.go:778] primary dev: ETH0 I1121 17:37:26.445700 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:37:26.445714 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:37:29.450407 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:37:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:37:29.450511 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:37:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:37:36.412253 3798194 memory.go:184] no items to output this cycle I1121 17:37:36.412401 3798194 cpu.go:275] no items to output this cycle E1121 17:37:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:37:46.411977 3798194 memory.go:184] no items to output this cycle I1121 17:37:46.412134 3798194 cpu.go:275] no items to output this cycle E1121 17:37:56.413333 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:37:56.413370 3798194 memory.go:191] Add success. W1121 17:37:56.413405 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:37:56.413417 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:37:56.413421 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:37:56.418240 3798194 cpu.go:282] Add success. I1121 17:37:56.441655 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:37:56.442155 3798194 net.go:642] Add success. I1121 17:37:56.443721 3798194 net.go:778] primary dev: ETH0 I1121 17:37:56.443736 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:37:56.443749 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:37:57.446469 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:37:57.446732 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:37:57.446744 3798194 disk_worker.go:723] disk space is not compliant W1121 17:37:57.446748 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:37:57.450365 3798194 disk_worker.go:509] system disk:vda1 I1121 17:37:57.450401 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:37:58.446009 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:37:58.468204 3798194 disk_info.go:125] begin check local disk info of client I1121 17:37:58.472148 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:37:58.472156 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047f8c0 0xc00047f900] I1121 17:37:59.450440 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:37:59.450520 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:37:59.450546 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:37:59.471132 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:38:06.412802 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:38:06.412824 3798194 memory.go:184] no items to output this cycle I1121 17:38:06.412971 3798194 cpu.go:275] no items to output this cycle E1121 17:38:16.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:38:16.411988 3798194 memory.go:184] no items to output this cycle I1121 17:38:16.412145 3798194 cpu.go:275] no items to output this cycle E1121 17:38:26.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:38:26.412264 3798194 memory.go:191] Add success. I1121 17:38:26.416469 3798194 cpu.go:282] Add success. I1121 17:38:26.441974 3798194 net.go:642] Add success. I1121 17:38:26.443487 3798194 net.go:778] primary dev: ETH0 I1121 17:38:26.443502 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:38:26.443515 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:38:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:38:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:38:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:38:36.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:38:36.412874 3798194 memory.go:184] no items to output this cycle I1121 17:38:36.413015 3798194 cpu.go:275] no items to output this cycle E1121 17:38:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:38:46.412259 3798194 memory.go:184] no items to output this cycle I1121 17:38:46.412407 3798194 cpu.go:275] no items to output this cycle E1121 17:38:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:38:56.412265 3798194 memory.go:191] Add success. W1121 17:38:56.412298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:38:56.412310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:38:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:38:56.416556 3798194 cpu.go:282] Add success. I1121 17:38:56.441918 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:38:56.442478 3798194 net.go:642] Add success. I1121 17:38:56.444128 3798194 net.go:778] primary dev: ETH0 I1121 17:38:56.444142 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:38:56.444153 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:38:57.455216 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:38:57.455386 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:38:57.455440 3798194 disk_worker.go:723] disk space is not compliant W1121 17:38:57.455443 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:38:57.466840 3798194 disk_worker.go:509] system disk:vda1 I1121 17:38:57.466886 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:38:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:38:58.473212 3798194 disk_info.go:125] begin check local disk info of client I1121 17:38:58.476734 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:38:58.476743 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f2b00 0xc0003f2b40] I1121 17:38:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:38:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:38:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:38:59.471686 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:39:06.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:39:06.413252 3798194 memory.go:184] no items to output this cycle I1121 17:39:06.413392 3798194 cpu.go:275] no items to output this cycle E1121 17:39:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:39:16.411988 3798194 memory.go:184] no items to output this cycle I1121 17:39:16.412151 3798194 cpu.go:275] no items to output this cycle I1121 17:39:17.527482 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:39:17.527490 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:39:26.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:39:26.412899 3798194 memory.go:191] Add success. I1121 17:39:26.417124 3798194 cpu.go:282] Add success. I1121 17:39:26.441301 3798194 net.go:642] Add success. I1121 17:39:26.442826 3798194 net.go:778] primary dev: ETH0 I1121 17:39:26.442841 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:39:26.442854 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:39:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:39:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:39:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:39:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:39:36.411974 3798194 memory.go:184] no items to output this cycle I1121 17:39:36.412126 3798194 cpu.go:275] no items to output this cycle E1121 17:39:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:39:46.411981 3798194 memory.go:184] no items to output this cycle I1121 17:39:46.412147 3798194 cpu.go:275] no items to output this cycle E1121 17:39:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:39:56.411957 3798194 memory.go:191] Add success. W1121 17:39:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:39:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:39:56.412008 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:39:56.412355 3798194 cpu.go:282] Add success. I1121 17:39:56.441803 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:39:56.442323 3798194 net.go:642] Add success. I1121 17:39:56.446146 3798194 net.go:778] primary dev: ETH0 I1121 17:39:56.446161 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:39:56.446175 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:39:57.450446 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:39:57.450660 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:39:57.450670 3798194 disk_worker.go:723] disk space is not compliant W1121 17:39:57.450673 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:39:57.459035 3798194 disk_worker.go:509] system disk:vda1 I1121 17:39:57.459083 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:39:58.446192 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:39:58.477437 3798194 disk_info.go:125] begin check local disk info of client I1121 17:39:58.480999 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:39:58.481007 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000310740 0xc000310780] I1121 17:39:59.449935 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:39:59.450015 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:39:59.450042 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:39:59.471585 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:40:06.412765 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:40:06.412787 3798194 memory.go:184] no items to output this cycle I1121 17:40:06.412938 3798194 cpu.go:275] no items to output this cycle E1121 17:40:16.412842 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:40:16.412868 3798194 memory.go:184] no items to output this cycle I1121 17:40:16.413012 3798194 cpu.go:275] no items to output this cycle E1121 17:40:26.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:40:26.412888 3798194 memory.go:191] Add success. I1121 17:40:26.417138 3798194 cpu.go:282] Add success. I1121 17:40:26.441662 3798194 net.go:642] Add success. I1121 17:40:26.443079 3798194 net.go:778] primary dev: ETH0 I1121 17:40:26.443094 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:40:26.443107 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:40:29.453132 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:40:29.453206 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:40:29.453231 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:40:36.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:40:36.411977 3798194 memory.go:184] no items to output this cycle I1121 17:40:36.412130 3798194 cpu.go:275] no items to output this cycle E1121 17:40:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:40:46.411986 3798194 memory.go:184] no items to output this cycle I1121 17:40:46.412143 3798194 cpu.go:275] no items to output this cycle E1121 17:40:56.412016 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:40:56.412056 3798194 memory.go:191] Add success. W1121 17:40:56.412091 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:40:56.412103 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:40:56.412107 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:40:56.412731 3798194 cpu.go:282] Add success. I1121 17:40:56.443918 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:40:56.444541 3798194 net.go:642] Add success. I1121 17:40:56.448678 3798194 net.go:778] primary dev: ETH0 I1121 17:40:56.448693 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:40:56.448706 3798194 net.go:659] Add eni metrics success, count: 7 E1121 17:40:57.451018 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 17:40:57.451032 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:40:57.451272 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:40:57.451282 3798194 disk_worker.go:723] disk space is not compliant W1121 17:40:57.451285 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:40:57.459851 3798194 disk_worker.go:509] system disk:vda1 I1121 17:40:57.459897 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 17:40:58.448503 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 17:40:58.448517 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:40:58.481767 3798194 disk_info.go:125] begin check local disk info of client I1121 17:40:58.485444 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:40:58.485453 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e080 0xc00030e0c0] E1121 17:40:59.448926 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 17:40:59.450002 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:40:59.450050 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:40:59.450070 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:40:59.471604 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:41:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:41:06.412261 3798194 memory.go:184] no items to output this cycle I1121 17:41:06.412407 3798194 cpu.go:275] no items to output this cycle E1121 17:41:16.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:41:16.412266 3798194 memory.go:184] no items to output this cycle I1121 17:41:16.412419 3798194 cpu.go:275] no items to output this cycle E1121 17:41:26.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:41:26.412275 3798194 memory.go:191] Add success. I1121 17:41:26.416814 3798194 cpu.go:282] Add success. I1121 17:41:26.441737 3798194 net.go:642] Add success. I1121 17:41:26.443378 3798194 net.go:778] primary dev: ETH0 I1121 17:41:26.443396 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:41:26.443408 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:41:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:41:29.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:41:29.450511 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:41:36.412259 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:41:36.412280 3798194 memory.go:184] no items to output this cycle I1121 17:41:36.412426 3798194 cpu.go:275] no items to output this cycle E1121 17:41:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:41:46.411974 3798194 memory.go:184] no items to output this cycle I1121 17:41:46.412130 3798194 cpu.go:275] no items to output this cycle E1121 17:41:56.411929 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:41:56.411962 3798194 memory.go:191] Add success. W1121 17:41:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:41:56.412005 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:41:56.412008 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:41:56.412360 3798194 cpu.go:282] Add success. I1121 17:41:56.441803 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:41:56.442333 3798194 net.go:642] Add success. I1121 17:41:56.446710 3798194 net.go:778] primary dev: ETH0 I1121 17:41:56.446725 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:41:56.446742 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:41:57.450813 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:41:57.451087 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:41:57.451101 3798194 disk_worker.go:723] disk space is not compliant W1121 17:41:57.451105 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:41:57.463887 3798194 disk_worker.go:509] system disk:vda1 I1121 17:41:57.463937 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:41:58.445985 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:41:58.486202 3798194 disk_info.go:125] begin check local disk info of client I1121 17:41:58.489757 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:41:58.489766 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f42c0 0xc0003f4300] I1121 17:41:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:41:59.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:41:59.450148 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:41:59.470682 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:42:06.412248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:42:06.412271 3798194 memory.go:184] no items to output this cycle I1121 17:42:06.412416 3798194 cpu.go:275] no items to output this cycle E1121 17:42:16.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:42:16.413260 3798194 memory.go:184] no items to output this cycle I1121 17:42:16.413407 3798194 cpu.go:275] no items to output this cycle I1121 17:42:17.527659 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:42:17.527667 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:42:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:42:26.412268 3798194 memory.go:191] Add success. I1121 17:42:26.416613 3798194 cpu.go:282] Add success. I1121 17:42:26.442228 3798194 net.go:642] Add success. I1121 17:42:26.443776 3798194 net.go:778] primary dev: ETH0 I1121 17:42:26.443792 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:42:26.443804 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:42:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:42:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:42:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:42:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:42:36.413254 3798194 memory.go:184] no items to output this cycle I1121 17:42:36.413400 3798194 cpu.go:275] no items to output this cycle E1121 17:42:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:42:46.411978 3798194 memory.go:184] no items to output this cycle I1121 17:42:46.412139 3798194 cpu.go:275] no items to output this cycle E1121 17:42:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:42:56.412271 3798194 memory.go:191] Add success. W1121 17:42:56.412304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:42:56.412316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:42:56.412319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:42:56.416998 3798194 cpu.go:282] Add success. I1121 17:42:56.441415 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:42:56.441891 3798194 net.go:642] Add success. I1121 17:42:56.443584 3798194 net.go:778] primary dev: ETH0 I1121 17:42:56.443599 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:42:56.443611 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:42:57.450644 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:42:57.450810 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:42:57.450866 3798194 disk_worker.go:723] disk space is not compliant W1121 17:42:57.450870 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:42:57.458972 3798194 disk_worker.go:509] system disk:vda1 I1121 17:42:57.459014 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:42:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:42:58.490198 3798194 disk_info.go:125] begin check local disk info of client I1121 17:42:58.493718 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:42:58.493726 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00038d740 0xc00038d780] I1121 17:42:59.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:42:59.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:42:59.450508 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:42:59.471026 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:43:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:43:06.412261 3798194 memory.go:184] no items to output this cycle I1121 17:43:06.412401 3798194 cpu.go:275] no items to output this cycle E1121 17:43:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:43:16.411979 3798194 memory.go:184] no items to output this cycle I1121 17:43:16.412146 3798194 cpu.go:275] no items to output this cycle E1121 17:43:26.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:43:26.412882 3798194 memory.go:191] Add success. I1121 17:43:26.417255 3798194 cpu.go:282] Add success. I1121 17:43:26.442043 3798194 net.go:642] Add success. I1121 17:43:26.443588 3798194 net.go:778] primary dev: ETH0 I1121 17:43:26.443608 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:43:26.443623 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:43:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:43:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:43:29.450501 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:43:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:43:36.411979 3798194 memory.go:184] no items to output this cycle I1121 17:43:36.412128 3798194 cpu.go:275] no items to output this cycle E1121 17:43:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:43:46.411974 3798194 memory.go:184] no items to output this cycle I1121 17:43:46.412133 3798194 cpu.go:275] no items to output this cycle E1121 17:43:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:43:56.412264 3798194 memory.go:191] Add success. W1121 17:43:56.412294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:43:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:43:56.412309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:43:56.416795 3798194 cpu.go:282] Add success. I1121 17:43:56.441328 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:43:56.441815 3798194 net.go:642] Add success. I1121 17:43:56.443346 3798194 net.go:778] primary dev: ETH0 I1121 17:43:56.443360 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:43:56.443372 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:43:57.450336 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:43:57.450505 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:43:57.450558 3798194 disk_worker.go:723] disk space is not compliant W1121 17:43:57.450561 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:43:57.458675 3798194 disk_worker.go:509] system disk:vda1 I1121 17:43:57.458720 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:43:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:43:58.494334 3798194 disk_info.go:125] begin check local disk info of client I1121 17:43:58.497874 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:43:58.497883 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00038d900 0xc00038d940] I1121 17:43:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:43:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:43:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:43:59.471657 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:44:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:44:06.411978 3798194 memory.go:184] no items to output this cycle I1121 17:44:06.412144 3798194 cpu.go:275] no items to output this cycle E1121 17:44:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:44:16.411976 3798194 memory.go:184] no items to output this cycle I1121 17:44:16.412130 3798194 cpu.go:275] no items to output this cycle E1121 17:44:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:44:26.411955 3798194 memory.go:191] Add success. I1121 17:44:26.412303 3798194 cpu.go:282] Add success. I1121 17:44:26.441978 3798194 net.go:642] Add success. I1121 17:44:26.446152 3798194 net.go:778] primary dev: ETH0 I1121 17:44:26.446169 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:44:26.446181 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:44:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:44:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:44:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:44:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:44:36.411981 3798194 memory.go:184] no items to output this cycle I1121 17:44:36.412135 3798194 cpu.go:275] no items to output this cycle E1121 17:44:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:44:46.411979 3798194 memory.go:184] no items to output this cycle I1121 17:44:46.412135 3798194 cpu.go:275] no items to output this cycle E1121 17:44:56.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:44:56.412275 3798194 memory.go:191] Add success. W1121 17:44:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:44:56.412314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:44:56.412317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:44:56.417170 3798194 cpu.go:282] Add success. I1121 17:44:56.441252 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:44:56.441715 3798194 net.go:642] Add success. I1121 17:44:56.443341 3798194 net.go:778] primary dev: ETH0 I1121 17:44:56.443355 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:44:56.443367 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:44:57.447927 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:44:57.448197 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:44:57.448209 3798194 disk_worker.go:723] disk space is not compliant W1121 17:44:57.448212 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:44:57.450957 3798194 disk_worker.go:509] system disk:vda1 I1121 17:44:57.451000 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:44:58.445640 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:44:58.497950 3798194 disk_info.go:125] begin check local disk info of client I1121 17:44:58.501471 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:44:58.501480 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ed40 0xc00030ed80] I1121 17:44:59.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:44:59.450131 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:44:59.450158 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:44:59.471647 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:45:06.413255 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:45:06.413284 3798194 memory.go:184] no items to output this cycle I1121 17:45:06.413435 3798194 cpu.go:275] no items to output this cycle E1121 17:45:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:45:16.411976 3798194 memory.go:184] no items to output this cycle I1121 17:45:16.412135 3798194 cpu.go:275] no items to output this cycle I1121 17:45:17.527820 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:45:17.527829 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:45:26.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:45:26.412266 3798194 memory.go:191] Add success. I1121 17:45:26.416868 3798194 cpu.go:282] Add success. I1121 17:45:26.441367 3798194 net.go:642] Add success. I1121 17:45:26.442838 3798194 net.go:778] primary dev: ETH0 I1121 17:45:26.442852 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:45:26.442864 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:45:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:45:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:45:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:45:36.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:45:36.413256 3798194 memory.go:184] no items to output this cycle I1121 17:45:36.413400 3798194 cpu.go:275] no items to output this cycle E1121 17:45:46.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:45:46.412878 3798194 memory.go:184] no items to output this cycle I1121 17:45:46.413025 3798194 cpu.go:275] no items to output this cycle E1121 17:45:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:45:56.411943 3798194 memory.go:191] Add success. W1121 17:45:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:45:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:45:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:45:56.412343 3798194 cpu.go:282] Add success. I1121 17:45:56.441560 3798194 event_worker.go:152] Polling the log file for events... I1121 17:45:56.444415 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:45:56.444917 3798194 net.go:642] Add success. I1121 17:45:56.448731 3798194 net.go:778] primary dev: ETH0 I1121 17:45:56.448746 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:45:56.448759 3798194 net.go:659] Add eni metrics success, count: 7 E1121 17:45:57.450888 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 17:45:57.450902 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:45:57.451069 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:45:57.451166 3798194 disk_worker.go:723] disk space is not compliant W1121 17:45:57.451169 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:45:57.465240 3798194 disk_worker.go:509] system disk:vda1 I1121 17:45:57.465292 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 17:45:58.448179 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 17:45:58.448194 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:45:58.502549 3798194 disk_info.go:125] begin check local disk info of client I1121 17:45:58.506037 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:45:58.506045 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fa340 0xc0003fa380] E1121 17:45:59.448627 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 17:45:59.449659 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:45:59.449710 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:45:59.449735 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:45:59.471265 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:46:06.412738 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:46:06.412762 3798194 memory.go:184] no items to output this cycle I1121 17:46:06.412912 3798194 cpu.go:275] no items to output this cycle E1121 17:46:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:46:16.411976 3798194 memory.go:184] no items to output this cycle I1121 17:46:16.412129 3798194 cpu.go:275] no items to output this cycle E1121 17:46:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:46:26.411946 3798194 memory.go:191] Add success. I1121 17:46:26.412307 3798194 cpu.go:282] Add success. I1121 17:46:26.441906 3798194 net.go:642] Add success. I1121 17:46:26.446209 3798194 net.go:778] primary dev: ETH0 I1121 17:46:26.446224 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:46:26.446237 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:46:29.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:46:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:46:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:46:36.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:46:36.412882 3798194 memory.go:184] no items to output this cycle I1121 17:46:36.413027 3798194 cpu.go:275] no items to output this cycle E1121 17:46:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:46:46.411976 3798194 memory.go:184] no items to output this cycle I1121 17:46:46.412133 3798194 cpu.go:275] no items to output this cycle E1121 17:46:56.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:46:56.413270 3798194 memory.go:191] Add success. W1121 17:46:56.413301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:46:56.413313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:46:56.413317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:46:56.417946 3798194 cpu.go:282] Add success. I1121 17:46:56.441413 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:46:56.441881 3798194 net.go:642] Add success. I1121 17:46:56.443532 3798194 net.go:778] primary dev: ETH0 I1121 17:46:56.443546 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:46:56.443557 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:46:57.447905 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:46:57.448127 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:46:57.448185 3798194 disk_worker.go:723] disk space is not compliant W1121 17:46:57.448188 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:46:57.450910 3798194 disk_worker.go:509] system disk:vda1 I1121 17:46:57.450963 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:46:58.446385 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:46:58.506776 3798194 disk_info.go:125] begin check local disk info of client I1121 17:46:58.510452 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:46:58.510461 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000381800 0xc000381840] I1121 17:46:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:46:59.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:46:59.450506 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:46:59.471020 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:47:06.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:47:06.412248 3798194 memory.go:184] no items to output this cycle I1121 17:47:06.412397 3798194 cpu.go:275] no items to output this cycle E1121 17:47:16.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:47:16.411990 3798194 memory.go:184] no items to output this cycle I1121 17:47:16.412144 3798194 cpu.go:275] no items to output this cycle E1121 17:47:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:47:26.413267 3798194 memory.go:191] Add success. I1121 17:47:26.417546 3798194 cpu.go:282] Add success. I1121 17:47:26.442406 3798194 net.go:642] Add success. I1121 17:47:26.443852 3798194 net.go:778] primary dev: ETH0 I1121 17:47:26.443868 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:47:26.443888 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:47:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:47:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:47:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:47:36.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:47:36.411976 3798194 memory.go:184] no items to output this cycle I1121 17:47:36.412129 3798194 cpu.go:275] no items to output this cycle E1121 17:47:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:47:46.413258 3798194 memory.go:184] no items to output this cycle I1121 17:47:46.413406 3798194 cpu.go:275] no items to output this cycle E1121 17:47:56.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:47:56.411959 3798194 memory.go:191] Add success. W1121 17:47:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:47:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:47:56.412006 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:47:56.412339 3798194 cpu.go:282] Add success. I1121 17:47:56.441811 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:47:56.442312 3798194 net.go:642] Add success. I1121 17:47:56.446668 3798194 net.go:778] primary dev: ETH0 I1121 17:47:56.446683 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:47:56.446697 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:47:57.450300 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:47:57.450519 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:47:57.450532 3798194 disk_worker.go:723] disk space is not compliant W1121 17:47:57.450535 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:47:57.458657 3798194 disk_worker.go:509] system disk:vda1 I1121 17:47:57.458712 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:47:58.446009 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:47:58.511434 3798194 disk_info.go:125] begin check local disk info of client I1121 17:47:58.514940 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:47:58.514947 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ffc00 0xc0003ffc40] I1121 17:47:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:47:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:47:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:47:59.471677 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:48:06.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:48:06.412252 3798194 memory.go:184] no items to output this cycle I1121 17:48:06.412397 3798194 cpu.go:275] no items to output this cycle E1121 17:48:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:48:16.411977 3798194 memory.go:184] no items to output this cycle I1121 17:48:16.412128 3798194 cpu.go:275] no items to output this cycle I1121 17:48:17.528016 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:48:17.528024 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:48:26.411665 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:48:26.411697 3798194 memory.go:191] Add success. I1121 17:48:26.412042 3798194 cpu.go:282] Add success. I1121 17:48:26.441680 3798194 net.go:642] Add success. I1121 17:48:26.445805 3798194 net.go:778] primary dev: ETH0 I1121 17:48:26.445822 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:48:26.445835 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:48:29.450029 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:48:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:48:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:48:36.412672 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:48:36.412695 3798194 memory.go:184] no items to output this cycle I1121 17:48:36.412836 3798194 cpu.go:275] no items to output this cycle E1121 17:48:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:48:46.411982 3798194 memory.go:184] no items to output this cycle I1121 17:48:46.412137 3798194 cpu.go:275] no items to output this cycle E1121 17:48:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:48:56.411951 3798194 memory.go:191] Add success. W1121 17:48:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:48:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:48:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:48:56.412329 3798194 cpu.go:282] Add success. I1121 17:48:56.441967 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:48:56.442984 3798194 net.go:642] Add success. I1121 17:48:56.447107 3798194 net.go:778] primary dev: ETH0 I1121 17:48:56.447135 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:48:56.447153 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:48:57.450300 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:48:57.450511 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:48:57.450525 3798194 disk_worker.go:723] disk space is not compliant W1121 17:48:57.450528 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:48:57.458615 3798194 disk_worker.go:509] system disk:vda1 I1121 17:48:57.458657 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:48:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:48:58.515202 3798194 disk_info.go:125] begin check local disk info of client I1121 17:48:58.518743 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:48:58.518752 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004e4180 0xc0004e41c0] I1121 17:48:59.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:48:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:48:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:48:59.471686 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:49:06.413254 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:49:06.413278 3798194 memory.go:184] no items to output this cycle I1121 17:49:06.413433 3798194 cpu.go:275] no items to output this cycle E1121 17:49:16.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:49:16.411986 3798194 memory.go:184] no items to output this cycle I1121 17:49:16.412165 3798194 cpu.go:275] no items to output this cycle E1121 17:49:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:49:26.411956 3798194 memory.go:191] Add success. I1121 17:49:26.412330 3798194 cpu.go:282] Add success. I1121 17:49:26.441914 3798194 net.go:642] Add success. I1121 17:49:26.449734 3798194 net.go:778] primary dev: ETH0 I1121 17:49:26.449750 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:49:26.449763 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:49:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:49:29.450121 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:49:29.450153 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:49:36.413123 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:49:36.413147 3798194 memory.go:184] no items to output this cycle I1121 17:49:36.413293 3798194 cpu.go:275] no items to output this cycle E1121 17:49:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:49:46.411978 3798194 memory.go:184] no items to output this cycle I1121 17:49:46.412132 3798194 cpu.go:275] no items to output this cycle E1121 17:49:56.411930 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:49:56.411962 3798194 memory.go:191] Add success. W1121 17:49:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:49:56.412009 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:49:56.412012 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:49:56.412398 3798194 cpu.go:282] Add success. I1121 17:49:56.441922 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:49:56.442506 3798194 net.go:642] Add success. I1121 17:49:56.446626 3798194 net.go:778] primary dev: ETH0 I1121 17:49:56.446641 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:49:56.446654 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:49:57.450336 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:49:57.450551 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:49:57.450561 3798194 disk_worker.go:723] disk space is not compliant W1121 17:49:57.450565 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:49:57.458559 3798194 disk_worker.go:509] system disk:vda1 I1121 17:49:57.458604 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:49:58.446663 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:49:58.518832 3798194 disk_info.go:125] begin check local disk info of client I1121 17:49:58.522448 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:49:58.522457 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036a7c0 0xc00036a800] I1121 17:49:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:49:59.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:49:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:49:59.471671 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:50:06.412771 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:50:06.412794 3798194 memory.go:184] no items to output this cycle I1121 17:50:06.413170 3798194 cpu.go:275] no items to output this cycle E1121 17:50:16.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:50:16.411970 3798194 memory.go:184] no items to output this cycle I1121 17:50:16.412134 3798194 cpu.go:275] no items to output this cycle E1121 17:50:26.412277 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:50:26.412324 3798194 memory.go:191] Add success. I1121 17:50:26.416822 3798194 cpu.go:282] Add success. I1121 17:50:26.441600 3798194 net.go:642] Add success. I1121 17:50:26.443393 3798194 net.go:778] primary dev: ETH0 I1121 17:50:26.443410 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:50:26.443423 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:50:29.450403 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:50:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:50:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:50:36.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:50:36.412249 3798194 memory.go:184] no items to output this cycle I1121 17:50:36.412391 3798194 cpu.go:275] no items to output this cycle E1121 17:50:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:50:46.411983 3798194 memory.go:184] no items to output this cycle I1121 17:50:46.412135 3798194 cpu.go:275] no items to output this cycle E1121 17:50:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:50:56.411951 3798194 memory.go:191] Add success. W1121 17:50:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:50:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:50:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:50:56.412335 3798194 cpu.go:282] Add success. I1121 17:50:56.445354 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:50:56.445842 3798194 net.go:642] Add success. I1121 17:50:56.449904 3798194 net.go:778] primary dev: ETH0 I1121 17:50:56.449919 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:50:56.449933 3798194 net.go:659] Add eni metrics success, count: 7 E1121 17:50:57.451973 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 17:50:57.451987 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:50:57.452167 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:50:57.452219 3798194 disk_worker.go:723] disk space is not compliant W1121 17:50:57.452222 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:50:57.461294 3798194 disk_worker.go:509] system disk:vda1 I1121 17:50:57.461335 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 17:50:58.448148 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 17:50:58.448163 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:50:58.522555 3798194 disk_info.go:125] begin check local disk info of client I1121 17:50:58.525935 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:50:58.525944 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037cf80 0xc00037cfc0] E1121 17:50:59.448606 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 17:50:59.449680 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:50:59.449729 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:50:59.449748 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:50:59.471265 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:51:06.412291 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:51:06.412321 3798194 memory.go:184] no items to output this cycle I1121 17:51:06.412524 3798194 cpu.go:275] no items to output this cycle E1121 17:51:16.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:51:16.412257 3798194 memory.go:184] no items to output this cycle I1121 17:51:16.412407 3798194 cpu.go:275] no items to output this cycle I1121 17:51:17.528197 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:51:17.528206 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:51:26.411374 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:51:26.411408 3798194 memory.go:191] Add success. I1121 17:51:26.411744 3798194 cpu.go:282] Add success. I1121 17:51:26.441545 3798194 net.go:642] Add success. I1121 17:51:26.446092 3798194 net.go:778] primary dev: ETH0 I1121 17:51:26.446123 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:51:26.446137 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:51:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:51:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:51:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:51:36.412419 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:51:36.412447 3798194 memory.go:184] no items to output this cycle I1121 17:51:36.412591 3798194 cpu.go:275] no items to output this cycle E1121 17:51:46.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:51:46.412261 3798194 memory.go:184] no items to output this cycle I1121 17:51:46.412406 3798194 cpu.go:275] no items to output this cycle E1121 17:51:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:51:56.411953 3798194 memory.go:191] Add success. W1121 17:51:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:51:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:51:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:51:56.412375 3798194 cpu.go:282] Add success. I1121 17:51:56.441836 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:51:56.442439 3798194 net.go:642] Add success. I1121 17:51:56.446474 3798194 net.go:778] primary dev: ETH0 I1121 17:51:56.446488 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:51:56.446502 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:51:57.450275 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:51:57.450482 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:51:57.450492 3798194 disk_worker.go:723] disk space is not compliant W1121 17:51:57.450495 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:51:57.458600 3798194 disk_worker.go:509] system disk:vda1 I1121 17:51:57.458645 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:51:58.445997 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:51:58.526492 3798194 disk_info.go:125] begin check local disk info of client I1121 17:51:58.529993 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:51:58.530002 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000436d80 0xc000436dc0] I1121 17:51:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:51:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:51:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:51:59.471670 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:52:06.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:52:06.412256 3798194 memory.go:184] no items to output this cycle I1121 17:52:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 17:52:16.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:52:16.413268 3798194 memory.go:184] no items to output this cycle I1121 17:52:16.413417 3798194 cpu.go:275] no items to output this cycle E1121 17:52:26.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:52:26.412276 3798194 memory.go:191] Add success. I1121 17:52:26.416553 3798194 cpu.go:282] Add success. I1121 17:52:26.442369 3798194 net.go:642] Add success. I1121 17:52:26.443969 3798194 net.go:778] primary dev: ETH0 I1121 17:52:26.443985 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:52:26.443999 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:52:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:52:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:52:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:52:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:52:36.411980 3798194 memory.go:184] no items to output this cycle I1121 17:52:36.412134 3798194 cpu.go:275] no items to output this cycle E1121 17:52:46.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:52:46.413262 3798194 memory.go:184] no items to output this cycle I1121 17:52:46.413408 3798194 cpu.go:275] no items to output this cycle E1121 17:52:56.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:52:56.413266 3798194 memory.go:191] Add success. W1121 17:52:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:52:56.413315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:52:56.413319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:52:56.417670 3798194 cpu.go:282] Add success. I1121 17:52:56.440975 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:52:56.441713 3798194 net.go:642] Add success. I1121 17:52:56.443330 3798194 net.go:778] primary dev: ETH0 I1121 17:52:56.443347 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:52:56.443365 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:52:57.450743 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:52:57.450957 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:52:57.450968 3798194 disk_worker.go:723] disk space is not compliant W1121 17:52:57.450972 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:52:57.459900 3798194 disk_worker.go:509] system disk:vda1 I1121 17:52:57.459948 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:52:58.446008 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:52:58.530524 3798194 disk_info.go:125] begin check local disk info of client I1121 17:52:58.534038 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:52:58.534046 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071200 0xc000071240] I1121 17:52:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:52:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:52:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:52:59.471665 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:53:06.411975 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:53:06.412000 3798194 memory.go:184] no items to output this cycle I1121 17:53:06.412162 3798194 cpu.go:275] no items to output this cycle E1121 17:53:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:53:16.412262 3798194 memory.go:184] no items to output this cycle I1121 17:53:16.412417 3798194 cpu.go:275] no items to output this cycle E1121 17:53:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:53:26.411955 3798194 memory.go:191] Add success. I1121 17:53:26.412330 3798194 cpu.go:282] Add success. I1121 17:53:26.441953 3798194 net.go:642] Add success. I1121 17:53:26.446049 3798194 net.go:778] primary dev: ETH0 I1121 17:53:26.446067 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:53:26.446079 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:53:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:53:29.450118 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:53:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:53:36.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:53:36.412264 3798194 memory.go:184] no items to output this cycle I1121 17:53:36.412409 3798194 cpu.go:275] no items to output this cycle E1121 17:53:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:53:46.411972 3798194 memory.go:184] no items to output this cycle I1121 17:53:46.412128 3798194 cpu.go:275] no items to output this cycle E1121 17:53:56.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:53:56.413270 3798194 memory.go:191] Add success. W1121 17:53:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:53:56.413323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:53:56.413326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:53:56.418130 3798194 cpu.go:282] Add success. I1121 17:53:56.441555 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:53:56.442028 3798194 net.go:642] Add success. I1121 17:53:56.443669 3798194 net.go:778] primary dev: ETH0 I1121 17:53:56.443684 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:53:56.443697 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:53:57.445229 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:53:57.445519 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:53:57.445531 3798194 disk_worker.go:723] disk space is not compliant W1121 17:53:57.445535 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:53:57.449182 3798194 disk_worker.go:509] system disk:vda1 I1121 17:53:57.449224 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:53:58.446367 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:53:58.534903 3798194 disk_info.go:125] begin check local disk info of client I1121 17:53:58.538458 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:53:58.538466 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003cdf00 0xc0003cdf40] I1121 17:53:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:53:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:53:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:53:59.471668 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:54:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:54:06.411983 3798194 memory.go:184] no items to output this cycle I1121 17:54:06.412131 3798194 cpu.go:275] no items to output this cycle E1121 17:54:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:54:16.411984 3798194 memory.go:184] no items to output this cycle I1121 17:54:16.412142 3798194 cpu.go:275] no items to output this cycle I1121 17:54:17.528371 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:54:17.528379 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:54:26.412251 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:54:26.412288 3798194 memory.go:191] Add success. I1121 17:54:26.416799 3798194 cpu.go:282] Add success. I1121 17:54:26.441302 3798194 net.go:642] Add success. I1121 17:54:26.442912 3798194 net.go:778] primary dev: ETH0 I1121 17:54:26.442926 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:54:26.442941 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:54:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:54:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:54:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:54:36.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:54:36.411974 3798194 memory.go:184] no items to output this cycle I1121 17:54:36.412129 3798194 cpu.go:275] no items to output this cycle E1121 17:54:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:54:46.411975 3798194 memory.go:184] no items to output this cycle I1121 17:54:46.412140 3798194 cpu.go:275] no items to output this cycle E1121 17:54:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:54:56.413268 3798194 memory.go:191] Add success. W1121 17:54:56.413296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:54:56.413309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:54:56.413312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:54:56.417973 3798194 cpu.go:282] Add success. I1121 17:54:56.441327 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:54:56.441811 3798194 net.go:642] Add success. I1121 17:54:56.443338 3798194 net.go:778] primary dev: ETH0 I1121 17:54:56.443353 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:54:56.443366 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:54:57.447955 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:54:57.448215 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:54:57.448226 3798194 disk_worker.go:723] disk space is not compliant W1121 17:54:57.448229 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:54:57.450972 3798194 disk_worker.go:509] system disk:vda1 I1121 17:54:57.451012 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:54:58.445989 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:54:58.539518 3798194 disk_info.go:125] begin check local disk info of client I1121 17:54:58.543082 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:54:58.543091 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037c200 0xc00037c240] I1121 17:54:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:54:59.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:54:59.450147 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:54:59.471642 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:55:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:55:06.411984 3798194 memory.go:184] no items to output this cycle I1121 17:55:06.412138 3798194 cpu.go:275] no items to output this cycle E1121 17:55:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:55:16.411982 3798194 memory.go:184] no items to output this cycle I1121 17:55:16.412138 3798194 cpu.go:275] no items to output this cycle E1121 17:55:26.413264 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:55:26.413298 3798194 memory.go:191] Add success. I1121 17:55:26.417774 3798194 cpu.go:282] Add success. I1121 17:55:26.441221 3798194 net.go:642] Add success. I1121 17:55:26.442781 3798194 net.go:778] primary dev: ETH0 I1121 17:55:26.442796 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:55:26.442809 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:55:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:55:29.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:55:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:55:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:55:36.413259 3798194 memory.go:184] no items to output this cycle I1121 17:55:36.413404 3798194 cpu.go:275] no items to output this cycle E1121 17:55:46.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:55:46.412267 3798194 memory.go:184] no items to output this cycle I1121 17:55:46.412421 3798194 cpu.go:275] no items to output this cycle E1121 17:55:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:55:56.411950 3798194 memory.go:191] Add success. W1121 17:55:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:55:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:55:56.411991 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:55:56.412356 3798194 cpu.go:282] Add success. I1121 17:55:56.441578 3798194 event_worker.go:152] Polling the log file for events... I1121 17:55:56.444431 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:55:56.444926 3798194 net.go:642] Add success. I1121 17:55:56.448978 3798194 net.go:778] primary dev: ETH0 I1121 17:55:56.448992 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:55:56.449005 3798194 net.go:659] Add eni metrics success, count: 7 E1121 17:55:57.450429 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 17:55:57.450443 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:55:57.450593 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:55:57.450641 3798194 disk_worker.go:723] disk space is not compliant W1121 17:55:57.450645 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:55:57.458661 3798194 disk_worker.go:509] system disk:vda1 I1121 17:55:57.458703 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 17:55:58.447786 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 17:55:58.447803 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:55:58.543208 3798194 disk_info.go:125] begin check local disk info of client I1121 17:55:58.546812 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:55:58.546820 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037c4c0 0xc00037c500] I1121 17:55:59.450209 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:55:59.450274 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:55:59.450294 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:55:59.450416 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 17:55:59.470844 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:56:06.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:56:06.412267 3798194 memory.go:184] no items to output this cycle I1121 17:56:06.412414 3798194 cpu.go:275] no items to output this cycle E1121 17:56:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:56:16.411976 3798194 memory.go:184] no items to output this cycle I1121 17:56:16.412132 3798194 cpu.go:275] no items to output this cycle E1121 17:56:26.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:56:26.412893 3798194 memory.go:191] Add success. I1121 17:56:26.417383 3798194 cpu.go:282] Add success. I1121 17:56:26.441923 3798194 net.go:642] Add success. I1121 17:56:26.443549 3798194 net.go:778] primary dev: ETH0 I1121 17:56:26.443563 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:56:26.443578 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:56:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:56:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:56:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:56:36.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:56:36.412872 3798194 memory.go:184] no items to output this cycle I1121 17:56:36.413014 3798194 cpu.go:275] no items to output this cycle E1121 17:56:46.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:56:46.411990 3798194 memory.go:184] no items to output this cycle I1121 17:56:46.412151 3798194 cpu.go:275] no items to output this cycle E1121 17:56:56.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:56:56.413274 3798194 memory.go:191] Add success. W1121 17:56:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:56:56.413319 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:56:56.413323 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:56:56.417649 3798194 cpu.go:282] Add success. I1121 17:56:56.440966 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:56:56.441635 3798194 net.go:642] Add success. I1121 17:56:56.443191 3798194 net.go:778] primary dev: ETH0 I1121 17:56:56.443207 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:56:56.443220 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:56:57.450820 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:56:57.451219 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:56:57.451282 3798194 disk_worker.go:723] disk space is not compliant W1121 17:56:57.451285 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:56:57.461273 3798194 disk_worker.go:509] system disk:vda1 I1121 17:56:57.461312 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:56:58.446373 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:56:58.546919 3798194 disk_info.go:125] begin check local disk info of client I1121 17:56:58.550512 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:56:58.550521 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a56c0 0xc0004a5700] I1121 17:56:59.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:56:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:56:59.450503 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:56:59.471029 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:57:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:57:06.411985 3798194 memory.go:184] no items to output this cycle I1121 17:57:06.412142 3798194 cpu.go:275] no items to output this cycle E1121 17:57:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:57:16.411983 3798194 memory.go:184] no items to output this cycle I1121 17:57:16.412160 3798194 cpu.go:275] no items to output this cycle I1121 17:57:17.529269 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 17:57:17.529278 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 17:57:26.411390 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:57:26.411424 3798194 memory.go:191] Add success. I1121 17:57:26.411763 3798194 cpu.go:282] Add success. I1121 17:57:26.441286 3798194 net.go:642] Add success. I1121 17:57:26.445736 3798194 net.go:778] primary dev: ETH0 I1121 17:57:26.445751 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:57:26.445764 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:57:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:57:29.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:57:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:57:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:57:36.413263 3798194 memory.go:184] no items to output this cycle I1121 17:57:36.413409 3798194 cpu.go:275] no items to output this cycle E1121 17:57:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:57:46.411982 3798194 memory.go:184] no items to output this cycle I1121 17:57:46.412146 3798194 cpu.go:275] no items to output this cycle E1121 17:57:56.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:57:56.411957 3798194 memory.go:191] Add success. W1121 17:57:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:57:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:57:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:57:56.412346 3798194 cpu.go:282] Add success. I1121 17:57:56.441812 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:57:56.442393 3798194 net.go:642] Add success. I1121 17:57:56.446441 3798194 net.go:778] primary dev: ETH0 I1121 17:57:56.446456 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:57:56.446469 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:57:57.450232 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:57:57.450394 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:57:57.450447 3798194 disk_worker.go:723] disk space is not compliant W1121 17:57:57.450450 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:57:57.458597 3798194 disk_worker.go:509] system disk:vda1 I1121 17:57:57.458655 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:57:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:57:58.550577 3798194 disk_info.go:125] begin check local disk info of client I1121 17:57:58.554136 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:57:58.554144 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b8b40 0xc0003b8b80] I1121 17:57:59.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:57:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:57:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:57:59.471676 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:58:06.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:58:06.413254 3798194 memory.go:184] no items to output this cycle I1121 17:58:06.413397 3798194 cpu.go:275] no items to output this cycle E1121 17:58:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:58:16.412262 3798194 memory.go:184] no items to output this cycle I1121 17:58:16.412408 3798194 cpu.go:275] no items to output this cycle E1121 17:58:26.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:58:26.413283 3798194 memory.go:191] Add success. I1121 17:58:26.417951 3798194 cpu.go:282] Add success. I1121 17:58:26.441677 3798194 net.go:642] Add success. I1121 17:58:26.443177 3798194 net.go:778] primary dev: ETH0 I1121 17:58:26.443192 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:58:26.443205 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:58:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:58:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:58:29.450501 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:58:36.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:58:36.411982 3798194 memory.go:184] no items to output this cycle I1121 17:58:36.412143 3798194 cpu.go:275] no items to output this cycle E1121 17:58:46.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:58:46.412264 3798194 memory.go:184] no items to output this cycle I1121 17:58:46.412417 3798194 cpu.go:275] no items to output this cycle E1121 17:58:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:58:56.412271 3798194 memory.go:191] Add success. W1121 17:58:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:58:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:58:56.412317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:58:56.416648 3798194 cpu.go:282] Add success. I1121 17:58:56.441273 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:58:56.442161 3798194 net.go:642] Add success. I1121 17:58:56.443884 3798194 net.go:778] primary dev: ETH0 I1121 17:58:56.443899 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:58:56.443911 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:58:57.450767 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:58:57.450966 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:58:57.450977 3798194 disk_worker.go:723] disk space is not compliant W1121 17:58:57.450980 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:58:57.459219 3798194 disk_worker.go:509] system disk:vda1 I1121 17:58:57.459258 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:58:58.446096 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:58:58.554246 3798194 disk_info.go:125] begin check local disk info of client I1121 17:58:58.557759 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:58:58.557767 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002fd340 0xc0002fd380] I1121 17:58:59.450199 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:58:59.450273 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:58:59.450300 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:58:59.470733 3798194 disk_local_worker.go:436] Get disk info: [] E1121 17:59:06.413560 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:59:06.413583 3798194 memory.go:184] no items to output this cycle I1121 17:59:06.413733 3798194 cpu.go:275] no items to output this cycle E1121 17:59:16.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:59:16.412250 3798194 memory.go:184] no items to output this cycle I1121 17:59:16.412395 3798194 cpu.go:275] no items to output this cycle E1121 17:59:26.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:59:26.413272 3798194 memory.go:191] Add success. I1121 17:59:26.417586 3798194 cpu.go:282] Add success. I1121 17:59:26.442122 3798194 net.go:642] Add success. I1121 17:59:26.443618 3798194 net.go:778] primary dev: ETH0 I1121 17:59:26.443632 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:59:26.443644 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:59:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:59:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:59:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 17:59:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:59:36.412257 3798194 memory.go:184] no items to output this cycle I1121 17:59:36.412403 3798194 cpu.go:275] no items to output this cycle E1121 17:59:46.413289 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:59:46.413315 3798194 memory.go:184] no items to output this cycle I1121 17:59:46.413583 3798194 cpu.go:275] no items to output this cycle E1121 17:59:56.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 17:59:56.413277 3798194 memory.go:191] Add success. W1121 17:59:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 17:59:56.413317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 17:59:56.413320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 17:59:56.417691 3798194 cpu.go:282] Add success. I1121 17:59:56.441105 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 17:59:56.441592 3798194 net.go:642] Add success. I1121 17:59:56.443265 3798194 net.go:778] primary dev: ETH0 I1121 17:59:56.443280 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 17:59:56.443292 3798194 net.go:659] Add eni metrics success, count: 7 I1121 17:59:57.450387 3798194 custom_config.go:106] query custom config with name: gpu W1121 17:59:57.450554 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 17:59:57.450605 3798194 disk_worker.go:723] disk space is not compliant W1121 17:59:57.450608 3798194 disk_worker.go:743] disk inode is not compliant I1121 17:59:57.458766 3798194 disk_worker.go:509] system disk:vda1 I1121 17:59:57.458810 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 17:59:58.445681 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 17:59:58.557870 3798194 disk_info.go:125] begin check local disk info of client I1121 17:59:58.561400 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 17:59:58.561407 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00027ccc0 0xc00027cd00] I1121 17:59:59.450410 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 17:59:59.450487 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 17:59:59.450517 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 17:59:59.471051 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:00:06.412653 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:00:06.412676 3798194 memory.go:184] no items to output this cycle I1121 18:00:06.412820 3798194 cpu.go:275] no items to output this cycle E1121 18:00:16.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:00:16.413257 3798194 memory.go:184] no items to output this cycle I1121 18:00:16.413412 3798194 cpu.go:275] no items to output this cycle I1121 18:00:17.529437 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:00:17.529445 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:00:26.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:00:26.412268 3798194 memory.go:191] Add success. I1121 18:00:26.416763 3798194 cpu.go:282] Add success. I1121 18:00:26.441266 3798194 net.go:642] Add success. I1121 18:00:26.442856 3798194 net.go:778] primary dev: ETH0 I1121 18:00:26.442871 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:00:26.442884 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:00:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:00:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:00:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:00:36.412512 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:00:36.412535 3798194 memory.go:184] no items to output this cycle I1121 18:00:36.412677 3798194 cpu.go:275] no items to output this cycle E1121 18:00:46.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:00:46.412263 3798194 memory.go:184] no items to output this cycle I1121 18:00:46.412412 3798194 cpu.go:275] no items to output this cycle E1121 18:00:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:00:56.411956 3798194 memory.go:191] Add success. W1121 18:00:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:00:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:00:56.412006 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:00:56.412341 3798194 cpu.go:282] Add success. I1121 18:00:56.444469 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:00:56.444952 3798194 net.go:642] Add success. I1121 18:00:56.453415 3798194 net.go:778] primary dev: ETH0 I1121 18:00:56.453431 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:00:56.453445 3798194 net.go:659] Add eni metrics success, count: 7 E1121 18:00:57.450735 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 18:00:57.450749 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:00:57.450897 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:00:57.450959 3798194 disk_worker.go:723] disk space is not compliant W1121 18:00:57.450962 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:00:57.459144 3798194 disk_worker.go:509] system disk:vda1 I1121 18:00:57.459190 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 18:00:58.448204 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 18:00:58.448219 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:00:58.561503 3798194 disk_info.go:125] begin check local disk info of client I1121 18:00:58.564956 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:00:58.564965 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000478500 0xc000478540] E1121 18:00:59.449076 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 18:00:59.450125 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:00:59.450178 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:00:59.450197 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:00:59.471726 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:01:06.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:01:06.413267 3798194 memory.go:184] no items to output this cycle I1121 18:01:06.413434 3798194 cpu.go:275] no items to output this cycle E1121 18:01:16.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:01:16.412268 3798194 memory.go:184] no items to output this cycle I1121 18:01:16.412415 3798194 cpu.go:275] no items to output this cycle E1121 18:01:26.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:01:26.412274 3798194 memory.go:191] Add success. I1121 18:01:26.416721 3798194 cpu.go:282] Add success. I1121 18:01:26.442263 3798194 net.go:642] Add success. I1121 18:01:26.443735 3798194 net.go:778] primary dev: ETH0 I1121 18:01:26.443749 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:01:26.443762 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:01:29.450033 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:01:29.450128 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:01:29.450155 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:01:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:01:36.412255 3798194 memory.go:184] no items to output this cycle I1121 18:01:36.412402 3798194 cpu.go:275] no items to output this cycle E1121 18:01:46.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:01:46.413253 3798194 memory.go:184] no items to output this cycle I1121 18:01:46.413399 3798194 cpu.go:275] no items to output this cycle E1121 18:01:56.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:01:56.412272 3798194 memory.go:191] Add success. W1121 18:01:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:01:56.412318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:01:56.412321 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:01:56.417264 3798194 cpu.go:282] Add success. I1121 18:01:56.441663 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:01:56.442277 3798194 net.go:642] Add success. I1121 18:01:56.443756 3798194 net.go:778] primary dev: ETH0 I1121 18:01:56.443771 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:01:56.443785 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:01:57.450838 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:01:57.450999 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:01:57.451053 3798194 disk_worker.go:723] disk space is not compliant W1121 18:01:57.451057 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:01:57.459265 3798194 disk_worker.go:509] system disk:vda1 I1121 18:01:57.459311 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:01:58.446381 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:01:58.565018 3798194 disk_info.go:125] begin check local disk info of client I1121 18:01:58.568561 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:01:58.568570 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003857c0 0xc000385800] I1121 18:01:59.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:01:59.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:01:59.450502 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:01:59.471028 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:02:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:02:06.411983 3798194 memory.go:184] no items to output this cycle I1121 18:02:06.412138 3798194 cpu.go:275] no items to output this cycle E1121 18:02:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:02:16.411982 3798194 memory.go:184] no items to output this cycle I1121 18:02:16.412144 3798194 cpu.go:275] no items to output this cycle E1121 18:02:26.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:02:26.412259 3798194 memory.go:191] Add success. I1121 18:02:26.416651 3798194 cpu.go:282] Add success. I1121 18:02:26.442181 3798194 net.go:642] Add success. I1121 18:02:26.443740 3798194 net.go:778] primary dev: ETH0 I1121 18:02:26.443754 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:02:26.443768 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:02:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:02:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:02:29.450145 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:02:36.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:02:36.412009 3798194 memory.go:184] no items to output this cycle I1121 18:02:36.412414 3798194 cpu.go:275] no items to output this cycle E1121 18:02:46.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:02:46.411999 3798194 memory.go:184] no items to output this cycle I1121 18:02:46.412163 3798194 cpu.go:275] no items to output this cycle E1121 18:02:56.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:02:56.412266 3798194 memory.go:191] Add success. W1121 18:02:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:02:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:02:56.412314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:02:56.416973 3798194 cpu.go:282] Add success. I1121 18:02:56.441371 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:02:56.441840 3798194 net.go:642] Add success. I1121 18:02:56.443506 3798194 net.go:778] primary dev: ETH0 I1121 18:02:56.443520 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:02:56.443532 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:02:57.450639 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:02:57.450821 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:02:57.450876 3798194 disk_worker.go:723] disk space is not compliant W1121 18:02:57.450879 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:02:57.459759 3798194 disk_worker.go:509] system disk:vda1 I1121 18:02:57.459817 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:02:58.446014 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:02:58.568674 3798194 disk_info.go:125] begin check local disk info of client I1121 18:02:58.572169 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:02:58.572178 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000470d00 0xc000470d40] I1121 18:02:59.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:02:59.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:02:59.450512 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:02:59.471069 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:03:06.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:03:06.411977 3798194 memory.go:184] no items to output this cycle I1121 18:03:06.412138 3798194 cpu.go:275] no items to output this cycle E1121 18:03:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:03:16.411976 3798194 memory.go:184] no items to output this cycle I1121 18:03:16.412131 3798194 cpu.go:275] no items to output this cycle I1121 18:03:17.530490 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:03:17.530499 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:03:26.411439 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:03:26.411473 3798194 memory.go:191] Add success. I1121 18:03:26.411803 3798194 cpu.go:282] Add success. I1121 18:03:26.441308 3798194 net.go:642] Add success. I1121 18:03:26.445335 3798194 net.go:778] primary dev: ETH0 I1121 18:03:26.445350 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:03:26.445366 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:03:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:03:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:03:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:03:36.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:03:36.412883 3798194 memory.go:184] no items to output this cycle I1121 18:03:36.413026 3798194 cpu.go:275] no items to output this cycle E1121 18:03:46.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:03:46.411985 3798194 memory.go:184] no items to output this cycle I1121 18:03:46.412148 3798194 cpu.go:275] no items to output this cycle E1121 18:03:56.411932 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:03:56.411967 3798194 memory.go:191] Add success. W1121 18:03:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:03:56.412012 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:03:56.412016 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:03:56.412369 3798194 cpu.go:282] Add success. I1121 18:03:56.441866 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:03:56.442469 3798194 net.go:642] Add success. I1121 18:03:56.446763 3798194 net.go:778] primary dev: ETH0 I1121 18:03:56.446780 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:03:56.446793 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:03:57.450343 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:03:57.450767 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:03:57.450779 3798194 disk_worker.go:723] disk space is not compliant W1121 18:03:57.450782 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:03:57.459644 3798194 disk_worker.go:509] system disk:vda1 I1121 18:03:57.459684 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:03:58.446019 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:03:58.572277 3798194 disk_info.go:125] begin check local disk info of client I1121 18:03:58.575829 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:03:58.575839 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004938c0 0xc000493900] I1121 18:03:59.450403 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:03:59.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:03:59.450512 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:03:59.471045 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:04:06.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:04:06.412258 3798194 memory.go:184] no items to output this cycle I1121 18:04:06.412403 3798194 cpu.go:275] no items to output this cycle E1121 18:04:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:04:16.411979 3798194 memory.go:184] no items to output this cycle I1121 18:04:16.412140 3798194 cpu.go:275] no items to output this cycle E1121 18:04:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:04:26.411954 3798194 memory.go:191] Add success. I1121 18:04:26.412306 3798194 cpu.go:282] Add success. I1121 18:04:26.441750 3798194 net.go:642] Add success. I1121 18:04:26.445775 3798194 net.go:778] primary dev: ETH0 I1121 18:04:26.445791 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:04:26.445805 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:04:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:04:29.450122 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:04:29.450149 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:04:36.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:04:36.413262 3798194 memory.go:184] no items to output this cycle I1121 18:04:36.413406 3798194 cpu.go:275] no items to output this cycle E1121 18:04:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:04:46.411975 3798194 memory.go:184] no items to output this cycle I1121 18:04:46.412135 3798194 cpu.go:275] no items to output this cycle E1121 18:04:56.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:04:56.413264 3798194 memory.go:191] Add success. W1121 18:04:56.413290 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:04:56.413302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:04:56.413305 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:04:56.417707 3798194 cpu.go:282] Add success. I1121 18:04:56.441092 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:04:56.441594 3798194 net.go:642] Add success. I1121 18:04:56.443269 3798194 net.go:778] primary dev: ETH0 I1121 18:04:56.443284 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:04:56.443296 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:04:57.450704 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:04:57.450866 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:04:57.450919 3798194 disk_worker.go:723] disk space is not compliant W1121 18:04:57.450922 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:04:57.459057 3798194 disk_worker.go:509] system disk:vda1 I1121 18:04:57.459101 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:04:58.446402 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:04:58.575939 3798194 disk_info.go:125] begin check local disk info of client I1121 18:04:58.579478 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:04:58.579488 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c4b00 0xc0003c4b40] I1121 18:04:59.450408 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:04:59.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:04:59.450511 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:04:59.471010 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:05:06.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:05:06.413266 3798194 memory.go:184] no items to output this cycle I1121 18:05:06.413422 3798194 cpu.go:275] no items to output this cycle E1121 18:05:16.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:05:16.411969 3798194 memory.go:184] no items to output this cycle I1121 18:05:16.412130 3798194 cpu.go:275] no items to output this cycle E1121 18:05:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:05:26.411951 3798194 memory.go:191] Add success. I1121 18:05:26.412472 3798194 cpu.go:282] Add success. I1121 18:05:26.441331 3798194 net.go:642] Add success. I1121 18:05:26.446087 3798194 net.go:778] primary dev: ETH0 I1121 18:05:26.446103 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:05:26.446131 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:05:29.450038 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:05:29.450129 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:05:29.450160 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:05:36.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:05:36.411981 3798194 memory.go:184] no items to output this cycle I1121 18:05:36.412141 3798194 cpu.go:275] no items to output this cycle E1121 18:05:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:05:46.411988 3798194 memory.go:184] no items to output this cycle I1121 18:05:46.412142 3798194 cpu.go:275] no items to output this cycle E1121 18:05:56.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:05:56.412889 3798194 memory.go:191] Add success. W1121 18:05:56.412922 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:05:56.412934 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:05:56.412937 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:05:56.417342 3798194 cpu.go:282] Add success. I1121 18:05:56.441511 3798194 event_worker.go:152] Polling the log file for events... I1121 18:05:56.444456 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:05:56.445015 3798194 net.go:642] Add success. I1121 18:05:56.446553 3798194 net.go:778] primary dev: ETH0 I1121 18:05:56.446567 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:05:56.446578 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:05:56.446604 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795253756, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 18:05:56.455922 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 18:05:57.445301 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 18:05:57.445320 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:05:57.445519 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:05:57.445609 3798194 disk_worker.go:723] disk space is not compliant W1121 18:05:57.445612 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:05:57.448047 3798194 disk_worker.go:509] system disk:vda1 I1121 18:05:57.448086 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 18:05:58.447826 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 18:05:58.447841 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:05:58.580513 3798194 disk_info.go:125] begin check local disk info of client I1121 18:05:58.584009 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:05:58.584017 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002fea80 0xc0002feac0] E1121 18:05:59.448619 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 18:05:59.449659 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:05:59.449709 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:05:59.449733 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:05:59.471190 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:06:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:06:06.411977 3798194 memory.go:184] no items to output this cycle I1121 18:06:06.412146 3798194 cpu.go:275] no items to output this cycle E1121 18:06:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:06:16.411981 3798194 memory.go:184] no items to output this cycle I1121 18:06:16.412140 3798194 cpu.go:275] no items to output this cycle I1121 18:06:17.530663 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:06:17.530672 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:06:26.411543 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:06:26.411577 3798194 memory.go:191] Add success. I1121 18:06:26.411923 3798194 cpu.go:282] Add success. I1121 18:06:26.441532 3798194 net.go:642] Add success. I1121 18:06:26.445648 3798194 net.go:778] primary dev: ETH0 I1121 18:06:26.445663 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:06:26.445678 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:06:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:06:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:06:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:06:36.412549 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:06:36.412573 3798194 memory.go:184] no items to output this cycle I1121 18:06:36.412720 3798194 cpu.go:275] no items to output this cycle E1121 18:06:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:06:46.411972 3798194 memory.go:184] no items to output this cycle I1121 18:06:46.412131 3798194 cpu.go:275] no items to output this cycle E1121 18:06:56.411997 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:06:56.412039 3798194 memory.go:191] Add success. W1121 18:06:56.412069 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:06:56.412081 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:06:56.412084 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:06:56.417261 3798194 cpu.go:282] Add success. I1121 18:06:56.441423 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:06:56.441890 3798194 net.go:642] Add success. I1121 18:06:56.443458 3798194 net.go:778] primary dev: ETH0 I1121 18:06:56.443472 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:06:56.443484 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:06:57.447960 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:06:57.448185 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:06:57.448242 3798194 disk_worker.go:723] disk space is not compliant W1121 18:06:57.448245 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:06:57.451033 3798194 disk_worker.go:509] system disk:vda1 I1121 18:06:57.451072 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:06:58.446386 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:06:58.584130 3798194 disk_info.go:125] begin check local disk info of client I1121 18:06:58.587675 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:06:58.587685 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e340 0xc00030e380] I1121 18:06:59.450406 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:06:59.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:06:59.450507 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:06:59.471023 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:07:06.421825 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:07:06.421855 3798194 memory.go:184] no items to output this cycle I1121 18:07:06.422069 3798194 cpu.go:275] no items to output this cycle E1121 18:07:16.412286 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:07:16.412317 3798194 memory.go:184] no items to output this cycle I1121 18:07:16.412523 3798194 cpu.go:275] no items to output this cycle E1121 18:07:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:07:26.411957 3798194 memory.go:191] Add success. I1121 18:07:26.412321 3798194 cpu.go:282] Add success. I1121 18:07:26.442046 3798194 net.go:642] Add success. I1121 18:07:26.446224 3798194 net.go:778] primary dev: ETH0 I1121 18:07:26.446240 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:07:26.446263 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:07:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:07:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:07:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:07:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:07:36.412258 3798194 memory.go:184] no items to output this cycle I1121 18:07:36.412404 3798194 cpu.go:275] no items to output this cycle E1121 18:07:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:07:46.411976 3798194 memory.go:184] no items to output this cycle I1121 18:07:46.412146 3798194 cpu.go:275] no items to output this cycle E1121 18:07:56.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:07:56.411962 3798194 memory.go:191] Add success. W1121 18:07:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:07:56.412007 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:07:56.412011 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:07:56.412366 3798194 cpu.go:282] Add success. I1121 18:07:56.441830 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:07:56.442350 3798194 net.go:642] Add success. I1121 18:07:56.446494 3798194 net.go:778] primary dev: ETH0 I1121 18:07:56.446509 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:07:56.446523 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:07:57.452027 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:07:57.452219 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:07:57.452278 3798194 disk_worker.go:723] disk space is not compliant W1121 18:07:57.452281 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:07:57.465460 3798194 disk_worker.go:509] system disk:vda1 I1121 18:07:57.465502 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:07:58.446375 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:07:58.587806 3798194 disk_info.go:125] begin check local disk info of client I1121 18:07:58.591370 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:07:58.591380 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041e0c0 0xc00041e100] I1121 18:07:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:07:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:07:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:07:59.471681 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:08:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:08:06.412254 3798194 memory.go:184] no items to output this cycle I1121 18:08:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 18:08:16.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:08:16.411990 3798194 memory.go:184] no items to output this cycle I1121 18:08:16.412153 3798194 cpu.go:275] no items to output this cycle E1121 18:08:26.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:08:26.411955 3798194 memory.go:191] Add success. I1121 18:08:26.412307 3798194 cpu.go:282] Add success. I1121 18:08:26.441941 3798194 net.go:642] Add success. I1121 18:08:26.446301 3798194 net.go:778] primary dev: ETH0 I1121 18:08:26.446317 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:08:26.446332 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:08:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:08:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:08:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:08:36.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:08:36.412268 3798194 memory.go:184] no items to output this cycle I1121 18:08:36.412418 3798194 cpu.go:275] no items to output this cycle E1121 18:08:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:08:46.411978 3798194 memory.go:184] no items to output this cycle I1121 18:08:46.412138 3798194 cpu.go:275] no items to output this cycle E1121 18:08:56.412248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:08:56.412282 3798194 memory.go:191] Add success. W1121 18:08:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:08:56.412325 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:08:56.412328 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:08:56.416641 3798194 cpu.go:282] Add success. I1121 18:08:56.440964 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:08:56.441551 3798194 net.go:642] Add success. I1121 18:08:56.443330 3798194 net.go:778] primary dev: ETH0 I1121 18:08:56.443344 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:08:56.443355 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:08:57.454823 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:08:57.455001 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:08:57.455057 3798194 disk_worker.go:723] disk space is not compliant W1121 18:08:57.455060 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:08:57.463745 3798194 disk_worker.go:509] system disk:vda1 I1121 18:08:57.463785 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:08:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:08:58.592199 3798194 disk_info.go:125] begin check local disk info of client I1121 18:08:58.595731 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:08:58.595743 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a1640 0xc0003a1680] I1121 18:08:59.450029 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:08:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:08:59.450153 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:08:59.471724 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:09:06.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:09:06.412265 3798194 memory.go:184] no items to output this cycle I1121 18:09:06.412422 3798194 cpu.go:275] no items to output this cycle E1121 18:09:16.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:09:16.412259 3798194 memory.go:184] no items to output this cycle I1121 18:09:16.412404 3798194 cpu.go:275] no items to output this cycle I1121 18:09:17.531520 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:09:17.531528 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:09:26.411217 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:09:26.411251 3798194 memory.go:191] Add success. I1121 18:09:26.411599 3798194 cpu.go:282] Add success. I1121 18:09:26.442213 3798194 net.go:642] Add success. I1121 18:09:26.446363 3798194 net.go:778] primary dev: ETH0 I1121 18:09:26.446391 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:09:26.446406 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:09:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:09:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:09:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:09:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:09:36.412253 3798194 memory.go:184] no items to output this cycle I1121 18:09:36.412394 3798194 cpu.go:275] no items to output this cycle E1121 18:09:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:09:46.411981 3798194 memory.go:184] no items to output this cycle I1121 18:09:46.412138 3798194 cpu.go:275] no items to output this cycle E1121 18:09:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:09:56.411948 3798194 memory.go:191] Add success. W1121 18:09:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:09:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:09:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:09:56.412349 3798194 cpu.go:282] Add success. I1121 18:09:56.441815 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:09:56.442382 3798194 net.go:642] Add success. I1121 18:09:56.446239 3798194 net.go:778] primary dev: ETH0 I1121 18:09:56.446255 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:09:56.446267 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:09:57.450248 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:09:57.450421 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:09:57.450472 3798194 disk_worker.go:723] disk space is not compliant W1121 18:09:57.450475 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:09:57.458478 3798194 disk_worker.go:509] system disk:vda1 I1121 18:09:57.458520 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:09:58.446023 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:09:58.598156 3798194 disk_info.go:125] begin check local disk info of client I1121 18:09:58.615412 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:09:58.615422 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00026c400 0xc00026c440] I1121 18:09:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:09:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:09:59.450140 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:09:59.471672 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:10:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:10:06.412260 3798194 memory.go:184] no items to output this cycle I1121 18:10:06.412403 3798194 cpu.go:275] no items to output this cycle E1121 18:10:16.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:10:16.412260 3798194 memory.go:184] no items to output this cycle I1121 18:10:16.412402 3798194 cpu.go:275] no items to output this cycle E1121 18:10:26.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:10:26.411950 3798194 memory.go:191] Add success. I1121 18:10:26.412296 3798194 cpu.go:282] Add success. I1121 18:10:26.441745 3798194 net.go:642] Add success. I1121 18:10:26.445765 3798194 net.go:778] primary dev: ETH0 I1121 18:10:26.445781 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:10:26.445794 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:10:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:10:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:10:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:10:36.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:10:36.412261 3798194 memory.go:184] no items to output this cycle I1121 18:10:36.412405 3798194 cpu.go:275] no items to output this cycle E1121 18:10:46.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:10:46.412260 3798194 memory.go:184] no items to output this cycle I1121 18:10:46.412415 3798194 cpu.go:275] no items to output this cycle E1121 18:10:56.411910 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:10:56.411942 3798194 memory.go:191] Add success. W1121 18:10:56.411967 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:10:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:10:56.411982 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:10:56.412321 3798194 cpu.go:282] Add success. I1121 18:10:56.444496 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:10:56.445471 3798194 net.go:642] Add success. I1121 18:10:56.449868 3798194 net.go:778] primary dev: ETH0 I1121 18:10:56.449886 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:10:56.449901 3798194 net.go:659] Add eni metrics success, count: 7 E1121 18:10:57.445607 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 18:10:57.445624 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:10:57.445834 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:10:57.445895 3798194 disk_worker.go:723] disk space is not compliant W1121 18:10:57.445898 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:10:57.448038 3798194 disk_worker.go:509] system disk:vda1 I1121 18:10:57.448078 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 18:10:58.448492 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 18:10:58.448508 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:10:58.616207 3798194 disk_info.go:125] begin check local disk info of client I1121 18:10:58.620072 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:10:58.620082 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004c3100 0xc0004c3140] E1121 18:10:59.448627 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 18:10:59.449659 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:10:59.449707 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:10:59.449732 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:10:59.471210 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:11:06.413191 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:11:06.413215 3798194 memory.go:184] no items to output this cycle I1121 18:11:06.413359 3798194 cpu.go:275] no items to output this cycle E1121 18:11:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:11:16.411974 3798194 memory.go:184] no items to output this cycle I1121 18:11:16.412131 3798194 cpu.go:275] no items to output this cycle E1121 18:11:26.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:11:26.412263 3798194 memory.go:191] Add success. I1121 18:11:26.416755 3798194 cpu.go:282] Add success. I1121 18:11:26.441242 3798194 net.go:642] Add success. I1121 18:11:26.442973 3798194 net.go:778] primary dev: ETH0 I1121 18:11:26.442996 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:11:26.443009 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:11:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:11:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:11:29.450126 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:11:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:11:36.411978 3798194 memory.go:184] no items to output this cycle I1121 18:11:36.412134 3798194 cpu.go:275] no items to output this cycle E1121 18:11:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:11:46.411977 3798194 memory.go:184] no items to output this cycle I1121 18:11:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 18:11:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:11:56.411953 3798194 memory.go:191] Add success. W1121 18:11:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:11:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:11:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:11:56.412356 3798194 cpu.go:282] Add success. I1121 18:11:56.441808 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:11:56.442403 3798194 net.go:642] Add success. I1121 18:11:56.446566 3798194 net.go:778] primary dev: ETH0 I1121 18:11:56.446581 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:11:56.446593 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:11:57.450283 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:11:57.450481 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:11:57.450493 3798194 disk_worker.go:723] disk space is not compliant W1121 18:11:57.450496 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:11:57.458643 3798194 disk_worker.go:509] system disk:vda1 I1121 18:11:57.458688 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:11:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:11:58.620155 3798194 disk_info.go:125] begin check local disk info of client I1121 18:11:58.634346 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:11:58.634354 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040c940 0xc00040c980] I1121 18:11:59.450516 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:11:59.450594 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:11:59.450620 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:11:59.471126 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:12:06.412758 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:12:06.412779 3798194 memory.go:184] no items to output this cycle I1121 18:12:06.413162 3798194 cpu.go:275] no items to output this cycle E1121 18:12:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:12:16.411976 3798194 memory.go:184] no items to output this cycle I1121 18:12:16.412143 3798194 cpu.go:275] no items to output this cycle I1121 18:12:17.532511 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:12:17.532520 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:12:26.411446 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:12:26.411482 3798194 memory.go:191] Add success. I1121 18:12:26.411818 3798194 cpu.go:282] Add success. I1121 18:12:26.441478 3798194 net.go:642] Add success. I1121 18:12:26.445384 3798194 net.go:778] primary dev: ETH0 I1121 18:12:26.445400 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:12:26.445423 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:12:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:12:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:12:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:12:36.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:12:36.412877 3798194 memory.go:184] no items to output this cycle I1121 18:12:36.413019 3798194 cpu.go:275] no items to output this cycle E1121 18:12:46.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:12:46.412265 3798194 memory.go:184] no items to output this cycle I1121 18:12:46.412411 3798194 cpu.go:275] no items to output this cycle E1121 18:12:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:12:56.412265 3798194 memory.go:191] Add success. W1121 18:12:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:12:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:12:56.412314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:12:56.416421 3798194 cpu.go:282] Add success. I1121 18:12:56.440951 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:12:56.441507 3798194 net.go:642] Add success. I1121 18:12:56.443298 3798194 net.go:778] primary dev: ETH0 I1121 18:12:56.443313 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:12:56.443326 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:12:57.450375 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:12:57.450607 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:12:57.450618 3798194 disk_worker.go:723] disk space is not compliant W1121 18:12:57.450621 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:12:57.458797 3798194 disk_worker.go:509] system disk:vda1 I1121 18:12:57.458839 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:12:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:12:58.635151 3798194 disk_info.go:125] begin check local disk info of client I1121 18:12:58.652474 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:12:58.652485 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037cf00 0xc00037cf40] I1121 18:12:59.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:12:59.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:12:59.450513 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:12:59.471035 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:13:06.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:13:06.412886 3798194 memory.go:184] no items to output this cycle I1121 18:13:06.413029 3798194 cpu.go:275] no items to output this cycle E1121 18:13:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:13:16.411978 3798194 memory.go:184] no items to output this cycle I1121 18:13:16.412133 3798194 cpu.go:275] no items to output this cycle E1121 18:13:26.411932 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:13:26.411965 3798194 memory.go:191] Add success. I1121 18:13:26.412295 3798194 cpu.go:282] Add success. I1121 18:13:26.441743 3798194 net.go:642] Add success. I1121 18:13:26.446211 3798194 net.go:778] primary dev: ETH0 I1121 18:13:26.446228 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:13:26.446241 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:13:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:13:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:13:29.450122 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:13:36.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:13:36.412264 3798194 memory.go:184] no items to output this cycle I1121 18:13:36.412407 3798194 cpu.go:275] no items to output this cycle E1121 18:13:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:13:46.411975 3798194 memory.go:184] no items to output this cycle I1121 18:13:46.412134 3798194 cpu.go:275] no items to output this cycle E1121 18:13:56.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:13:56.412262 3798194 memory.go:191] Add success. W1121 18:13:56.412290 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:13:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:13:56.412305 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:13:56.416933 3798194 cpu.go:282] Add success. I1121 18:13:56.441843 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:13:56.442448 3798194 net.go:642] Add success. I1121 18:13:56.444153 3798194 net.go:778] primary dev: ETH0 I1121 18:13:56.444168 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:13:56.444181 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:13:57.450294 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:13:57.450502 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:13:57.450514 3798194 disk_worker.go:723] disk space is not compliant W1121 18:13:57.450517 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:13:57.458554 3798194 disk_worker.go:509] system disk:vda1 I1121 18:13:57.458598 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:13:58.446410 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:13:58.653159 3798194 disk_info.go:125] begin check local disk info of client I1121 18:13:58.670547 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:13:58.670557 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000418dc0 0xc000418e00] I1121 18:13:59.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:13:59.450465 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:13:59.450490 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:13:59.471002 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:14:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:14:06.411984 3798194 memory.go:184] no items to output this cycle I1121 18:14:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 18:14:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:14:16.411980 3798194 memory.go:184] no items to output this cycle I1121 18:14:16.412135 3798194 cpu.go:275] no items to output this cycle E1121 18:14:26.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:14:26.413271 3798194 memory.go:191] Add success. I1121 18:14:26.417849 3798194 cpu.go:282] Add success. I1121 18:14:26.441285 3798194 net.go:642] Add success. I1121 18:14:26.442966 3798194 net.go:778] primary dev: ETH0 I1121 18:14:26.442983 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:14:26.442995 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:14:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:14:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:14:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:14:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:14:36.412260 3798194 memory.go:184] no items to output this cycle I1121 18:14:36.412400 3798194 cpu.go:275] no items to output this cycle E1121 18:14:46.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:14:46.412259 3798194 memory.go:184] no items to output this cycle I1121 18:14:46.412405 3798194 cpu.go:275] no items to output this cycle E1121 18:14:56.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:14:56.412266 3798194 memory.go:191] Add success. W1121 18:14:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:14:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:14:56.412314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:14:56.416665 3798194 cpu.go:282] Add success. I1121 18:14:56.440956 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:14:56.441666 3798194 net.go:642] Add success. I1121 18:14:56.443288 3798194 net.go:778] primary dev: ETH0 I1121 18:14:56.443303 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:14:56.443321 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:14:57.451023 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:14:57.451264 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:14:57.451275 3798194 disk_worker.go:723] disk space is not compliant W1121 18:14:57.451278 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:14:57.459417 3798194 disk_worker.go:509] system disk:vda1 I1121 18:14:57.459469 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:14:58.446196 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:14:58.671159 3798194 disk_info.go:125] begin check local disk info of client I1121 18:14:58.690431 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:14:58.690440 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004ba800 0xc0004ba840] I1121 18:14:59.450198 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:14:59.450274 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:14:59.450300 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:14:59.470826 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:15:06.413249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:15:06.413272 3798194 memory.go:184] no items to output this cycle I1121 18:15:06.413415 3798194 cpu.go:275] no items to output this cycle E1121 18:15:16.412010 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:15:16.412039 3798194 memory.go:184] no items to output this cycle I1121 18:15:16.412256 3798194 cpu.go:275] no items to output this cycle I1121 18:15:17.533281 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:15:17.533290 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:15:26.411398 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:15:26.411430 3798194 memory.go:191] Add success. I1121 18:15:26.411779 3798194 cpu.go:282] Add success. I1121 18:15:26.441297 3798194 net.go:642] Add success. I1121 18:15:26.445548 3798194 net.go:778] primary dev: ETH0 I1121 18:15:26.445564 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:15:26.445576 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:15:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:15:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:15:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:15:36.412881 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:15:36.412903 3798194 memory.go:184] no items to output this cycle I1121 18:15:36.413052 3798194 cpu.go:275] no items to output this cycle E1121 18:15:46.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:15:46.413256 3798194 memory.go:184] no items to output this cycle I1121 18:15:46.413400 3798194 cpu.go:275] no items to output this cycle E1121 18:15:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:15:56.411950 3798194 memory.go:191] Add success. W1121 18:15:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:15:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:15:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:15:56.412341 3798194 cpu.go:282] Add success. I1121 18:15:56.441587 3798194 event_worker.go:152] Polling the log file for events... I1121 18:15:56.444429 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:15:56.444902 3798194 net.go:642] Add success. I1121 18:15:56.449261 3798194 net.go:778] primary dev: ETH0 I1121 18:15:56.449280 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:15:56.449291 3798194 net.go:659] Add eni metrics success, count: 7 E1121 18:15:57.450388 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 18:15:57.450402 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:15:57.450560 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:15:57.450617 3798194 disk_worker.go:723] disk space is not compliant W1121 18:15:57.450620 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:15:57.458639 3798194 disk_worker.go:509] system disk:vda1 I1121 18:15:57.458681 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 18:15:58.447831 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 18:15:58.447846 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:15:58.691146 3798194 disk_info.go:125] begin check local disk info of client I1121 18:15:58.707393 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:15:58.707403 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00042e000 0xc00042e040] E1121 18:15:59.448605 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 18:15:59.449679 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:15:59.449728 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:15:59.449755 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:15:59.471270 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:16:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:16:06.412258 3798194 memory.go:184] no items to output this cycle I1121 18:16:06.412413 3798194 cpu.go:275] no items to output this cycle E1121 18:16:16.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:16:16.411985 3798194 memory.go:184] no items to output this cycle I1121 18:16:16.412272 3798194 cpu.go:275] no items to output this cycle E1121 18:16:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:16:26.411946 3798194 memory.go:191] Add success. I1121 18:16:26.412333 3798194 cpu.go:282] Add success. I1121 18:16:26.441933 3798194 net.go:642] Add success. I1121 18:16:26.445732 3798194 net.go:778] primary dev: ETH0 I1121 18:16:26.445749 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:16:26.445763 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:16:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:16:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:16:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:16:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:16:36.413256 3798194 memory.go:184] no items to output this cycle I1121 18:16:36.413399 3798194 cpu.go:275] no items to output this cycle E1121 18:16:46.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:16:46.413266 3798194 memory.go:184] no items to output this cycle I1121 18:16:46.413413 3798194 cpu.go:275] no items to output this cycle E1121 18:16:56.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:16:56.413271 3798194 memory.go:191] Add success. W1121 18:16:56.413301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:16:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:16:56.413315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:16:56.417575 3798194 cpu.go:282] Add success. I1121 18:16:56.440918 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:16:56.441524 3798194 net.go:642] Add success. I1121 18:16:56.443124 3798194 net.go:778] primary dev: ETH0 I1121 18:16:56.443139 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:16:56.443153 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:16:57.448851 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:16:57.449095 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:16:57.449108 3798194 disk_worker.go:723] disk space is not compliant W1121 18:16:57.449128 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:16:57.450916 3798194 disk_worker.go:509] system disk:vda1 I1121 18:16:57.450955 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:16:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:16:58.716150 3798194 disk_info.go:125] begin check local disk info of client I1121 18:16:58.726438 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:16:58.726446 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000327b80 0xc000327bc0] I1121 18:16:59.450350 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:16:59.450422 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:16:59.450451 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:16:59.470948 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:17:06.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:17:06.413259 3798194 memory.go:184] no items to output this cycle I1121 18:17:06.413400 3798194 cpu.go:275] no items to output this cycle E1121 18:17:16.413297 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:17:16.413319 3798194 memory.go:184] no items to output this cycle I1121 18:17:16.413468 3798194 cpu.go:275] no items to output this cycle E1121 18:17:26.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:17:26.412263 3798194 memory.go:191] Add success. I1121 18:17:26.424147 3798194 cpu.go:282] Add success. I1121 18:17:26.441918 3798194 net.go:642] Add success. I1121 18:17:26.443399 3798194 net.go:778] primary dev: ETH0 I1121 18:17:26.443413 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:17:26.443426 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:17:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:17:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:17:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:17:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:17:36.412256 3798194 memory.go:184] no items to output this cycle I1121 18:17:36.412401 3798194 cpu.go:275] no items to output this cycle E1121 18:17:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:17:46.413267 3798194 memory.go:184] no items to output this cycle I1121 18:17:46.413427 3798194 cpu.go:275] no items to output this cycle E1121 18:17:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:17:56.412273 3798194 memory.go:191] Add success. W1121 18:17:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:17:56.412320 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:17:56.412323 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:17:56.416511 3798194 cpu.go:282] Add success. I1121 18:17:56.440970 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:17:56.441549 3798194 net.go:642] Add success. I1121 18:17:56.443107 3798194 net.go:778] primary dev: ETH0 I1121 18:17:56.443133 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:17:56.443145 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:17:57.450701 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:17:57.450935 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:17:57.450946 3798194 disk_worker.go:723] disk space is not compliant W1121 18:17:57.450949 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:17:57.459359 3798194 disk_worker.go:509] system disk:vda1 I1121 18:17:57.459399 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:17:58.446373 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:17:58.727160 3798194 disk_info.go:125] begin check local disk info of client I1121 18:17:58.744454 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:17:58.744463 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e14c0 0xc0002e1500] I1121 18:17:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:17:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:17:59.450504 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:17:59.471042 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:18:06.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:18:06.412266 3798194 memory.go:184] no items to output this cycle I1121 18:18:06.412413 3798194 cpu.go:275] no items to output this cycle E1121 18:18:16.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:18:16.412265 3798194 memory.go:184] no items to output this cycle I1121 18:18:16.412416 3798194 cpu.go:275] no items to output this cycle I1121 18:18:17.533452 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:18:17.533462 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:18:26.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:18:26.413275 3798194 memory.go:191] Add success. I1121 18:18:26.417775 3798194 cpu.go:282] Add success. I1121 18:18:26.441449 3798194 net.go:642] Add success. I1121 18:18:26.443008 3798194 net.go:778] primary dev: ETH0 I1121 18:18:26.443022 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:18:26.443034 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:18:29.449668 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:18:29.449740 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:18:29.449765 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:18:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:18:36.413258 3798194 memory.go:184] no items to output this cycle I1121 18:18:36.413418 3798194 cpu.go:275] no items to output this cycle E1121 18:18:46.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:18:46.411968 3798194 memory.go:184] no items to output this cycle I1121 18:18:46.412135 3798194 cpu.go:275] no items to output this cycle E1121 18:18:56.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:18:56.413275 3798194 memory.go:191] Add success. W1121 18:18:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:18:56.413317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:18:56.413321 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:18:56.417711 3798194 cpu.go:282] Add success. I1121 18:18:56.441060 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:18:56.441558 3798194 net.go:642] Add success. I1121 18:18:56.443003 3798194 net.go:778] primary dev: ETH0 I1121 18:18:56.443017 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:18:56.443031 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:18:57.450654 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:18:57.450841 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:18:57.450913 3798194 disk_worker.go:723] disk space is not compliant W1121 18:18:57.450917 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:18:57.460018 3798194 disk_worker.go:509] system disk:vda1 I1121 18:18:57.460060 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:18:58.446374 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:18:58.751158 3798194 disk_info.go:125] begin check local disk info of client I1121 18:18:58.768420 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:18:58.768429 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004567c0 0xc000456800] I1121 18:18:59.450486 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:18:59.450565 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:18:59.450592 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:18:59.471128 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:19:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:19:06.413255 3798194 memory.go:184] no items to output this cycle I1121 18:19:06.413396 3798194 cpu.go:275] no items to output this cycle E1121 18:19:16.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:19:16.411992 3798194 memory.go:184] no items to output this cycle I1121 18:19:16.412151 3798194 cpu.go:275] no items to output this cycle E1121 18:19:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:19:26.413264 3798194 memory.go:191] Add success. I1121 18:19:26.417453 3798194 cpu.go:282] Add success. I1121 18:19:26.441975 3798194 net.go:642] Add success. I1121 18:19:26.443376 3798194 net.go:778] primary dev: ETH0 I1121 18:19:26.443393 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:19:26.443408 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:19:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:19:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:19:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:19:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:19:36.411980 3798194 memory.go:184] no items to output this cycle I1121 18:19:36.412133 3798194 cpu.go:275] no items to output this cycle E1121 18:19:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:19:46.411977 3798194 memory.go:184] no items to output this cycle I1121 18:19:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 18:19:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:19:56.412264 3798194 memory.go:191] Add success. W1121 18:19:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:19:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:19:56.412312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:19:56.416402 3798194 cpu.go:282] Add success. I1121 18:19:56.441217 3798194 net.go:642] Add success. I1121 18:19:56.442980 3798194 net.go:778] primary dev: ETH0 I1121 18:19:56.442995 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:19:56.443007 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:19:56.441553 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:19:57.445284 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:19:57.445503 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:19:57.445587 3798194 disk_worker.go:723] disk space is not compliant W1121 18:19:57.445591 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:19:57.448059 3798194 disk_worker.go:509] system disk:vda1 I1121 18:19:57.448100 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:19:58.445992 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:19:58.769154 3798194 disk_info.go:125] begin check local disk info of client I1121 18:19:58.776496 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:19:58.776506 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ca0c0 0xc0003ca100] I1121 18:19:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:19:59.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:19:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:19:59.471651 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:20:06.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:20:06.411971 3798194 memory.go:184] no items to output this cycle I1121 18:20:06.412138 3798194 cpu.go:275] no items to output this cycle E1121 18:20:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:20:16.411980 3798194 memory.go:184] no items to output this cycle I1121 18:20:16.412136 3798194 cpu.go:275] no items to output this cycle E1121 18:20:26.411933 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:20:26.411966 3798194 memory.go:191] Add success. I1121 18:20:26.412308 3798194 cpu.go:282] Add success. I1121 18:20:26.442102 3798194 net.go:642] Add success. I1121 18:20:26.446159 3798194 net.go:778] primary dev: ETH0 I1121 18:20:26.446176 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:20:26.446196 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:20:29.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:20:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:20:29.450152 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:20:36.412080 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:20:36.412103 3798194 memory.go:184] no items to output this cycle I1121 18:20:36.412282 3798194 cpu.go:275] no items to output this cycle E1121 18:20:46.413333 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:20:46.413355 3798194 memory.go:184] no items to output this cycle I1121 18:20:46.413498 3798194 cpu.go:275] no items to output this cycle E1121 18:20:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:20:56.413261 3798194 memory.go:191] Add success. W1121 18:20:56.413287 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:20:56.413300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:20:56.413304 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:20:56.417447 3798194 cpu.go:282] Add success. I1121 18:20:56.444381 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:20:56.444848 3798194 net.go:642] Add success. I1121 18:20:56.446320 3798194 net.go:778] primary dev: ETH0 I1121 18:20:56.446335 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:20:56.446348 3798194 net.go:659] Add eni metrics success, count: 7 E1121 18:20:57.450745 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 18:20:57.450759 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:20:57.450914 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:20:57.450980 3798194 disk_worker.go:723] disk space is not compliant W1121 18:20:57.450983 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:20:57.459211 3798194 disk_worker.go:509] system disk:vda1 I1121 18:20:57.459256 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 18:20:58.448202 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 18:20:58.448216 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:20:58.777159 3798194 disk_info.go:125] begin check local disk info of client I1121 18:20:58.794573 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:20:58.794583 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000272d80 0xc000272dc0] E1121 18:20:59.448607 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 18:20:59.449681 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:20:59.449731 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:20:59.449749 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:20:59.471181 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:21:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:21:06.411978 3798194 memory.go:184] no items to output this cycle I1121 18:21:06.412137 3798194 cpu.go:275] no items to output this cycle E1121 18:21:16.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:21:16.412891 3798194 memory.go:184] no items to output this cycle I1121 18:21:16.413045 3798194 cpu.go:275] no items to output this cycle I1121 18:21:17.534270 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:21:17.534279 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:21:26.411226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:21:26.411275 3798194 memory.go:191] Add success. I1121 18:21:26.411603 3798194 cpu.go:282] Add success. I1121 18:21:26.442250 3798194 net.go:642] Add success. I1121 18:21:26.446406 3798194 net.go:778] primary dev: ETH0 I1121 18:21:26.446423 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:21:26.446436 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:21:29.450410 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:21:29.450488 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:21:29.450514 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:21:36.413165 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:21:36.413188 3798194 memory.go:184] no items to output this cycle I1121 18:21:36.413325 3798194 cpu.go:275] no items to output this cycle E1121 18:21:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:21:46.411987 3798194 memory.go:184] no items to output this cycle I1121 18:21:46.412155 3798194 cpu.go:275] no items to output this cycle E1121 18:21:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:21:56.411948 3798194 memory.go:191] Add success. W1121 18:21:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:21:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:21:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:21:56.412349 3798194 cpu.go:282] Add success. I1121 18:21:56.441359 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:21:56.442252 3798194 net.go:642] Add success. I1121 18:21:56.446636 3798194 net.go:778] primary dev: ETH0 I1121 18:21:56.446651 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:21:56.446666 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:21:57.450335 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:21:57.450512 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:21:57.450570 3798194 disk_worker.go:723] disk space is not compliant W1121 18:21:57.450573 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:21:57.458675 3798194 disk_worker.go:509] system disk:vda1 I1121 18:21:57.458713 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:21:58.446600 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:21:58.795159 3798194 disk_info.go:125] begin check local disk info of client I1121 18:21:58.811501 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:21:58.811512 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004d9a00 0xc0004d9a40] I1121 18:21:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:21:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:21:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:21:59.471703 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:22:06.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:22:06.411990 3798194 memory.go:184] no items to output this cycle I1121 18:22:06.412144 3798194 cpu.go:275] no items to output this cycle E1121 18:22:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:22:16.411982 3798194 memory.go:184] no items to output this cycle I1121 18:22:16.412144 3798194 cpu.go:275] no items to output this cycle E1121 18:22:26.411888 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:22:26.411921 3798194 memory.go:191] Add success. I1121 18:22:26.412283 3798194 cpu.go:282] Add success. I1121 18:22:26.441912 3798194 net.go:642] Add success. I1121 18:22:26.445819 3798194 net.go:778] primary dev: ETH0 I1121 18:22:26.445835 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:22:26.445847 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:22:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:22:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:22:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:22:36.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:22:36.412251 3798194 memory.go:184] no items to output this cycle I1121 18:22:36.412393 3798194 cpu.go:275] no items to output this cycle E1121 18:22:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:22:46.411977 3798194 memory.go:184] no items to output this cycle I1121 18:22:46.412129 3798194 cpu.go:275] no items to output this cycle E1121 18:22:56.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:22:56.411959 3798194 memory.go:191] Add success. W1121 18:22:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:22:56.412001 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:22:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:22:56.412357 3798194 cpu.go:282] Add success. I1121 18:22:56.441802 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:22:56.442459 3798194 net.go:642] Add success. I1121 18:22:56.446604 3798194 net.go:778] primary dev: ETH0 I1121 18:22:56.446619 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:22:56.446631 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:22:57.450303 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:22:57.450465 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:22:57.450523 3798194 disk_worker.go:723] disk space is not compliant W1121 18:22:57.450526 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:22:57.458650 3798194 disk_worker.go:509] system disk:vda1 I1121 18:22:57.458695 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:22:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:22:58.812150 3798194 disk_info.go:125] begin check local disk info of client I1121 18:22:58.828557 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:22:58.828566 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041a940 0xc00041a980] I1121 18:22:59.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:22:59.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:22:59.450505 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:22:59.471024 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:23:06.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:23:06.412251 3798194 memory.go:184] no items to output this cycle I1121 18:23:06.412406 3798194 cpu.go:275] no items to output this cycle E1121 18:23:16.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:23:16.411989 3798194 memory.go:184] no items to output this cycle I1121 18:23:16.412419 3798194 cpu.go:275] no items to output this cycle E1121 18:23:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:23:26.413268 3798194 memory.go:191] Add success. I1121 18:23:26.417344 3798194 cpu.go:282] Add success. I1121 18:23:26.442022 3798194 net.go:642] Add success. I1121 18:23:26.443530 3798194 net.go:778] primary dev: ETH0 I1121 18:23:26.443544 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:23:26.443557 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:23:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:23:29.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:23:29.450153 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:23:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:23:36.411979 3798194 memory.go:184] no items to output this cycle I1121 18:23:36.412134 3798194 cpu.go:275] no items to output this cycle E1121 18:23:46.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:23:46.411988 3798194 memory.go:184] no items to output this cycle I1121 18:23:46.412147 3798194 cpu.go:275] no items to output this cycle E1121 18:23:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:23:56.411956 3798194 memory.go:191] Add success. W1121 18:23:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:23:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:23:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:23:56.412382 3798194 cpu.go:282] Add success. I1121 18:23:56.441904 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:23:56.442447 3798194 net.go:642] Add success. I1121 18:23:56.446244 3798194 net.go:778] primary dev: ETH0 I1121 18:23:56.446257 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:23:56.446271 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:23:57.450191 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:23:57.450401 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:23:57.450412 3798194 disk_worker.go:723] disk space is not compliant W1121 18:23:57.450415 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:23:57.458435 3798194 disk_worker.go:509] system disk:vda1 I1121 18:23:57.458477 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:23:58.446198 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:23:58.828671 3798194 disk_info.go:125] begin check local disk info of client I1121 18:23:58.832565 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:23:58.832574 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030db80 0xc00030dbc0] I1121 18:23:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:23:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:23:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:23:59.471678 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:24:06.412770 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:24:06.412793 3798194 memory.go:184] no items to output this cycle I1121 18:24:06.412942 3798194 cpu.go:275] no items to output this cycle E1121 18:24:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:24:16.411985 3798194 memory.go:184] no items to output this cycle I1121 18:24:16.412155 3798194 cpu.go:275] no items to output this cycle I1121 18:24:17.534442 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:24:17.534452 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:24:26.411294 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:24:26.411326 3798194 memory.go:191] Add success. I1121 18:24:26.411687 3798194 cpu.go:282] Add success. I1121 18:24:26.441296 3798194 net.go:642] Add success. I1121 18:24:26.445499 3798194 net.go:778] primary dev: ETH0 I1121 18:24:26.445515 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:24:26.445529 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:24:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:24:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:24:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:24:36.412342 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:24:36.412364 3798194 memory.go:184] no items to output this cycle I1121 18:24:36.412508 3798194 cpu.go:275] no items to output this cycle E1121 18:24:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:24:46.411974 3798194 memory.go:184] no items to output this cycle I1121 18:24:46.412128 3798194 cpu.go:275] no items to output this cycle E1121 18:24:56.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:24:56.411964 3798194 memory.go:191] Add success. W1121 18:24:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:24:56.412008 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:24:56.412012 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:24:56.412360 3798194 cpu.go:282] Add success. I1121 18:24:56.441808 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:24:56.442338 3798194 net.go:642] Add success. I1121 18:24:56.446677 3798194 net.go:778] primary dev: ETH0 I1121 18:24:56.446691 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:24:56.446705 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:24:57.450385 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:24:57.450607 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:24:57.450620 3798194 disk_worker.go:723] disk space is not compliant W1121 18:24:57.450623 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:24:57.458830 3798194 disk_worker.go:509] system disk:vda1 I1121 18:24:57.458873 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:24:58.445703 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:24:58.833157 3798194 disk_info.go:125] begin check local disk info of client I1121 18:24:58.850575 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:24:58.850585 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e14c0 0xc0002e1500] I1121 18:24:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:24:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:24:59.450133 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:24:59.471693 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:25:06.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:25:06.411992 3798194 memory.go:184] no items to output this cycle I1121 18:25:06.412149 3798194 cpu.go:275] no items to output this cycle E1121 18:25:16.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:25:16.413249 3798194 memory.go:184] no items to output this cycle I1121 18:25:16.413395 3798194 cpu.go:275] no items to output this cycle E1121 18:25:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:25:26.411946 3798194 memory.go:191] Add success. I1121 18:25:26.412315 3798194 cpu.go:282] Add success. I1121 18:25:26.442057 3798194 net.go:642] Add success. I1121 18:25:26.446145 3798194 net.go:778] primary dev: ETH0 I1121 18:25:26.446160 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:25:26.446172 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:25:29.450411 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:25:29.450490 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:25:29.450521 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:25:36.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:25:36.412874 3798194 memory.go:184] no items to output this cycle I1121 18:25:36.413018 3798194 cpu.go:275] no items to output this cycle E1121 18:25:46.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:25:46.411985 3798194 memory.go:184] no items to output this cycle I1121 18:25:46.412144 3798194 cpu.go:275] no items to output this cycle E1121 18:25:56.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:25:56.412894 3798194 memory.go:191] Add success. W1121 18:25:56.412925 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:25:56.412937 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:25:56.412941 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:25:56.417068 3798194 cpu.go:282] Add success. I1121 18:25:56.443888 3798194 event_worker.go:152] Polling the log file for events... I1121 18:25:56.444100 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:25:56.444614 3798194 net.go:642] Add success. I1121 18:25:56.446341 3798194 net.go:778] primary dev: ETH0 I1121 18:25:56.446355 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:25:56.446373 3798194 net.go:659] Add eni metrics success, count: 7 E1121 18:25:57.450389 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 18:25:57.450404 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:25:57.450610 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:25:57.450620 3798194 disk_worker.go:723] disk space is not compliant W1121 18:25:57.450623 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:25:57.458599 3798194 disk_worker.go:509] system disk:vda1 I1121 18:25:57.458641 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 18:25:58.447794 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 18:25:58.447809 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:25:58.851148 3798194 disk_info.go:125] begin check local disk info of client I1121 18:25:58.868575 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:25:58.868587 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e48c0 0xc0003e4900] E1121 18:25:59.448719 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 18:25:59.449793 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:25:59.449844 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:25:59.449864 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:25:59.471443 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:26:06.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:26:06.411974 3798194 memory.go:184] no items to output this cycle I1121 18:26:06.412124 3798194 cpu.go:275] no items to output this cycle E1121 18:26:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:26:16.412263 3798194 memory.go:184] no items to output this cycle I1121 18:26:16.412410 3798194 cpu.go:275] no items to output this cycle E1121 18:26:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:26:26.411948 3798194 memory.go:191] Add success. I1121 18:26:26.412297 3798194 cpu.go:282] Add success. I1121 18:26:26.442026 3798194 net.go:642] Add success. I1121 18:26:26.445999 3798194 net.go:778] primary dev: ETH0 I1121 18:26:26.446014 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:26:26.446027 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:26:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:26:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:26:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:26:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:26:36.412254 3798194 memory.go:184] no items to output this cycle I1121 18:26:36.412396 3798194 cpu.go:275] no items to output this cycle E1121 18:26:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:26:46.411974 3798194 memory.go:184] no items to output this cycle I1121 18:26:46.412130 3798194 cpu.go:275] no items to output this cycle E1121 18:26:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:26:56.411956 3798194 memory.go:191] Add success. W1121 18:26:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:26:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:26:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:26:56.412322 3798194 cpu.go:282] Add success. I1121 18:26:56.441877 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:26:56.442460 3798194 net.go:642] Add success. I1121 18:26:56.446453 3798194 net.go:778] primary dev: ETH0 I1121 18:26:56.446469 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:26:56.446481 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:26:57.450215 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:26:57.450410 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:26:57.450422 3798194 disk_worker.go:723] disk space is not compliant W1121 18:26:57.450426 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:26:57.458429 3798194 disk_worker.go:509] system disk:vda1 I1121 18:26:57.458468 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:26:58.446443 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:26:58.872160 3798194 disk_info.go:125] begin check local disk info of client I1121 18:26:58.889443 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:26:58.889454 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00035b480 0xc00035b4c0] I1121 18:26:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:26:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:26:59.450133 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:26:59.471677 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:27:06.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:27:06.413272 3798194 memory.go:184] no items to output this cycle I1121 18:27:06.413447 3798194 cpu.go:275] no items to output this cycle E1121 18:27:16.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:27:16.412256 3798194 memory.go:184] no items to output this cycle I1121 18:27:16.412399 3798194 cpu.go:275] no items to output this cycle I1121 18:27:17.534610 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:27:17.534619 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:27:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:27:26.413263 3798194 memory.go:191] Add success. I1121 18:27:26.417644 3798194 cpu.go:282] Add success. I1121 18:27:26.441270 3798194 net.go:642] Add success. I1121 18:27:26.442886 3798194 net.go:778] primary dev: ETH0 I1121 18:27:26.442902 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:27:26.442915 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:27:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:27:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:27:29.450501 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:27:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:27:36.411974 3798194 memory.go:184] no items to output this cycle I1121 18:27:36.412328 3798194 cpu.go:275] no items to output this cycle E1121 18:27:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:27:46.411974 3798194 memory.go:184] no items to output this cycle I1121 18:27:46.412139 3798194 cpu.go:275] no items to output this cycle E1121 18:27:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:27:56.412267 3798194 memory.go:191] Add success. W1121 18:27:56.412296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:27:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:27:56.412310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:27:56.416866 3798194 cpu.go:282] Add success. I1121 18:27:56.441265 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:27:56.441820 3798194 net.go:642] Add success. I1121 18:27:56.443396 3798194 net.go:778] primary dev: ETH0 I1121 18:27:56.443411 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:27:56.443424 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:27:57.450226 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:27:57.450389 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:27:57.450441 3798194 disk_worker.go:723] disk space is not compliant W1121 18:27:57.450444 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:27:57.458483 3798194 disk_worker.go:509] system disk:vda1 I1121 18:27:57.458523 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:27:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:27:58.895148 3798194 disk_info.go:125] begin check local disk info of client I1121 18:27:58.898669 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:27:58.898679 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00035aa80 0xc00035aac0] I1121 18:27:59.450010 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:27:59.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:27:59.450126 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:27:59.471659 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:28:06.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:28:06.411970 3798194 memory.go:184] no items to output this cycle I1121 18:28:06.412122 3798194 cpu.go:275] no items to output this cycle E1121 18:28:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:28:16.411981 3798194 memory.go:184] no items to output this cycle I1121 18:28:16.412137 3798194 cpu.go:275] no items to output this cycle E1121 18:28:26.413333 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:28:26.413367 3798194 memory.go:191] Add success. I1121 18:28:26.417486 3798194 cpu.go:282] Add success. I1121 18:28:26.441905 3798194 net.go:642] Add success. I1121 18:28:26.443358 3798194 net.go:778] primary dev: ETH0 I1121 18:28:26.443372 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:28:26.443386 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:28:29.450403 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:28:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:28:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:28:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:28:36.411977 3798194 memory.go:184] no items to output this cycle I1121 18:28:36.412123 3798194 cpu.go:275] no items to output this cycle E1121 18:28:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:28:46.411973 3798194 memory.go:184] no items to output this cycle I1121 18:28:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 18:28:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:28:56.411947 3798194 memory.go:191] Add success. W1121 18:28:56.411974 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:28:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:28:56.411989 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:28:56.412315 3798194 cpu.go:282] Add success. I1121 18:28:56.441765 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:28:56.442271 3798194 net.go:642] Add success. I1121 18:28:56.446155 3798194 net.go:778] primary dev: ETH0 I1121 18:28:56.446172 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:28:56.446183 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:28:57.450276 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:28:57.450442 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:28:57.450494 3798194 disk_worker.go:723] disk space is not compliant W1121 18:28:57.450498 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:28:57.459287 3798194 disk_worker.go:509] system disk:vda1 I1121 18:28:57.459327 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:28:58.446011 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:28:58.900146 3798194 disk_info.go:125] begin check local disk info of client I1121 18:28:58.903755 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:28:58.903764 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000491800 0xc000491840] I1121 18:28:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:28:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:28:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:28:59.471663 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:29:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:29:06.412257 3798194 memory.go:184] no items to output this cycle I1121 18:29:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 18:29:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:29:16.411981 3798194 memory.go:184] no items to output this cycle I1121 18:29:16.412135 3798194 cpu.go:275] no items to output this cycle E1121 18:29:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:29:26.411959 3798194 memory.go:191] Add success. I1121 18:29:26.412311 3798194 cpu.go:282] Add success. I1121 18:29:26.441752 3798194 net.go:642] Add success. I1121 18:29:26.446072 3798194 net.go:778] primary dev: ETH0 I1121 18:29:26.446089 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:29:26.446103 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:29:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:29:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:29:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:29:36.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:29:36.413268 3798194 memory.go:184] no items to output this cycle I1121 18:29:36.413413 3798194 cpu.go:275] no items to output this cycle E1121 18:29:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:29:46.411976 3798194 memory.go:184] no items to output this cycle I1121 18:29:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 18:29:56.413250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:29:56.413282 3798194 memory.go:191] Add success. W1121 18:29:56.413313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:29:56.413325 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:29:56.413328 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:29:56.417188 3798194 cpu.go:282] Add success. I1121 18:29:56.441632 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:29:56.442222 3798194 net.go:642] Add success. I1121 18:29:56.444007 3798194 net.go:778] primary dev: ETH0 I1121 18:29:56.444021 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:29:56.444034 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:29:57.449992 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:29:57.450170 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:29:57.450223 3798194 disk_worker.go:723] disk space is not compliant W1121 18:29:57.450226 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:29:57.458191 3798194 disk_worker.go:509] system disk:vda1 I1121 18:29:57.458233 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:29:58.446064 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:29:58.905303 3798194 disk_info.go:125] begin check local disk info of client I1121 18:29:58.908804 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:29:58.908813 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003daa80 0xc0003daac0] I1121 18:29:59.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:29:59.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:29:59.450497 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:29:59.471061 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:30:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:30:06.412257 3798194 memory.go:184] no items to output this cycle I1121 18:30:06.412406 3798194 cpu.go:275] no items to output this cycle E1121 18:30:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:30:16.411980 3798194 memory.go:184] no items to output this cycle I1121 18:30:16.412141 3798194 cpu.go:275] no items to output this cycle I1121 18:30:17.535268 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:30:17.535276 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:30:26.411227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:30:26.411260 3798194 memory.go:191] Add success. I1121 18:30:26.411733 3798194 cpu.go:282] Add success. I1121 18:30:26.441320 3798194 net.go:642] Add success. I1121 18:30:26.445330 3798194 net.go:778] primary dev: ETH0 I1121 18:30:26.445345 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:30:26.445358 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:30:29.450032 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:30:29.450118 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:30:29.450149 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:30:36.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:30:36.413251 3798194 memory.go:184] no items to output this cycle I1121 18:30:36.413397 3798194 cpu.go:275] no items to output this cycle E1121 18:30:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:30:46.413253 3798194 memory.go:184] no items to output this cycle I1121 18:30:46.413398 3798194 cpu.go:275] no items to output this cycle E1121 18:30:56.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:30:56.413259 3798194 memory.go:191] Add success. W1121 18:30:56.413286 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:30:56.413298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:30:56.413301 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:30:56.417487 3798194 cpu.go:282] Add success. I1121 18:30:56.442190 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:30:56.442769 3798194 net.go:642] Add success. I1121 18:30:56.445568 3798194 net.go:778] primary dev: ETH0 I1121 18:30:56.445583 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:30:56.445595 3798194 net.go:659] Add eni metrics success, count: 7 E1121 18:30:57.447912 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 18:30:57.447933 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:30:57.448140 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:30:57.448197 3798194 disk_worker.go:723] disk space is not compliant W1121 18:30:57.448200 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:30:57.451001 3798194 disk_worker.go:509] system disk:vda1 I1121 18:30:57.451044 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 18:30:58.447800 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 18:30:58.447814 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:30:58.908929 3798194 disk_info.go:125] begin check local disk info of client I1121 18:30:58.912552 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:30:58.912563 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002fe840 0xc0002fe880] E1121 18:30:59.448926 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 18:30:59.450004 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:30:59.450052 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:30:59.450076 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:30:59.471568 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:31:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:31:06.411978 3798194 memory.go:184] no items to output this cycle I1121 18:31:06.412130 3798194 cpu.go:275] no items to output this cycle E1121 18:31:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:31:16.411982 3798194 memory.go:184] no items to output this cycle I1121 18:31:16.412136 3798194 cpu.go:275] no items to output this cycle E1121 18:31:26.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:31:26.412273 3798194 memory.go:191] Add success. I1121 18:31:26.416442 3798194 cpu.go:282] Add success. I1121 18:31:26.441951 3798194 net.go:642] Add success. I1121 18:31:26.443311 3798194 net.go:778] primary dev: ETH0 I1121 18:31:26.443325 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:31:26.443338 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:31:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:31:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:31:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:31:36.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:31:36.412885 3798194 memory.go:184] no items to output this cycle I1121 18:31:36.413040 3798194 cpu.go:275] no items to output this cycle E1121 18:31:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:31:46.411971 3798194 memory.go:184] no items to output this cycle I1121 18:31:46.412129 3798194 cpu.go:275] no items to output this cycle E1121 18:31:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:31:56.411956 3798194 memory.go:191] Add success. W1121 18:31:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:31:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:31:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:31:56.412344 3798194 cpu.go:282] Add success. I1121 18:31:56.440967 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:31:56.441544 3798194 net.go:642] Add success. I1121 18:31:56.445776 3798194 net.go:778] primary dev: ETH0 I1121 18:31:56.445799 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:31:56.445812 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:31:57.450694 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:31:57.450910 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:31:57.450922 3798194 disk_worker.go:723] disk space is not compliant W1121 18:31:57.450925 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:31:57.459087 3798194 disk_worker.go:509] system disk:vda1 I1121 18:31:57.459146 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:31:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:31:58.914144 3798194 disk_info.go:125] begin check local disk info of client I1121 18:31:58.917648 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:31:58.917657 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e080 0xc00030e0c0] I1121 18:31:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:31:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:31:59.450130 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:31:59.471654 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:32:06.412249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:32:06.412272 3798194 memory.go:184] no items to output this cycle I1121 18:32:06.412414 3798194 cpu.go:275] no items to output this cycle E1121 18:32:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:32:16.411980 3798194 memory.go:184] no items to output this cycle I1121 18:32:16.412142 3798194 cpu.go:275] no items to output this cycle E1121 18:32:26.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:32:26.411951 3798194 memory.go:191] Add success. I1121 18:32:26.412312 3798194 cpu.go:282] Add success. I1121 18:32:26.441901 3798194 net.go:642] Add success. I1121 18:32:26.446193 3798194 net.go:778] primary dev: ETH0 I1121 18:32:26.446209 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:32:26.446221 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:32:29.450006 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:32:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:32:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:32:36.412372 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:32:36.412396 3798194 memory.go:184] no items to output this cycle I1121 18:32:36.412549 3798194 cpu.go:275] no items to output this cycle E1121 18:32:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:32:46.411975 3798194 memory.go:184] no items to output this cycle I1121 18:32:46.412127 3798194 cpu.go:275] no items to output this cycle E1121 18:32:56.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:32:56.413279 3798194 memory.go:191] Add success. W1121 18:32:56.413307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:32:56.413319 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:32:56.413322 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:32:56.417884 3798194 cpu.go:282] Add success. I1121 18:32:56.441419 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:32:56.441884 3798194 net.go:642] Add success. I1121 18:32:56.443434 3798194 net.go:778] primary dev: ETH0 I1121 18:32:56.443450 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:32:56.443462 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:32:57.447887 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:32:57.448086 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:32:57.448174 3798194 disk_worker.go:723] disk space is not compliant W1121 18:32:57.448178 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:32:57.450948 3798194 disk_worker.go:509] system disk:vda1 I1121 18:32:57.450995 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:32:58.446397 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:32:58.919777 3798194 disk_info.go:125] begin check local disk info of client I1121 18:32:58.923348 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:32:58.923357 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000448cc0 0xc000448d00] I1121 18:32:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:32:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:32:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:32:59.471657 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:33:06.412246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:33:06.412267 3798194 memory.go:184] no items to output this cycle I1121 18:33:06.412407 3798194 cpu.go:275] no items to output this cycle E1121 18:33:16.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:33:16.411993 3798194 memory.go:184] no items to output this cycle I1121 18:33:16.412163 3798194 cpu.go:275] no items to output this cycle I1121 18:33:17.535440 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:33:17.535449 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:33:26.411213 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:33:26.411248 3798194 memory.go:191] Add success. I1121 18:33:26.411614 3798194 cpu.go:282] Add success. I1121 18:33:26.441299 3798194 net.go:642] Add success. I1121 18:33:26.445163 3798194 net.go:778] primary dev: ETH0 I1121 18:33:26.445179 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:33:26.445192 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:33:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:33:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:33:29.450501 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:33:36.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:33:36.412246 3798194 memory.go:184] no items to output this cycle I1121 18:33:36.412389 3798194 cpu.go:275] no items to output this cycle E1121 18:33:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:33:46.411980 3798194 memory.go:184] no items to output this cycle I1121 18:33:46.412141 3798194 cpu.go:275] no items to output this cycle E1121 18:33:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:33:56.411956 3798194 memory.go:191] Add success. W1121 18:33:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:33:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:33:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:33:56.412330 3798194 cpu.go:282] Add success. I1121 18:33:56.441790 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:33:56.442304 3798194 net.go:642] Add success. I1121 18:33:56.446575 3798194 net.go:778] primary dev: ETH0 I1121 18:33:56.446590 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:33:56.446605 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:33:57.450313 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:33:57.450476 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:33:57.450529 3798194 disk_worker.go:723] disk space is not compliant W1121 18:33:57.450532 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:33:57.458762 3798194 disk_worker.go:509] system disk:vda1 I1121 18:33:57.458806 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:33:58.445995 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:33:58.923461 3798194 disk_info.go:125] begin check local disk info of client I1121 18:33:58.927331 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:33:58.927340 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037e200 0xc00037e240] I1121 18:33:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:33:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:33:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:33:59.471669 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:34:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:34:06.412257 3798194 memory.go:184] no items to output this cycle I1121 18:34:06.412400 3798194 cpu.go:275] no items to output this cycle E1121 18:34:16.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:34:16.411986 3798194 memory.go:184] no items to output this cycle I1121 18:34:16.412151 3798194 cpu.go:275] no items to output this cycle E1121 18:34:26.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:34:26.413264 3798194 memory.go:191] Add success. I1121 18:34:26.417657 3798194 cpu.go:282] Add success. I1121 18:34:26.442209 3798194 net.go:642] Add success. I1121 18:34:26.443745 3798194 net.go:778] primary dev: ETH0 I1121 18:34:26.443760 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:34:26.443773 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:34:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:34:29.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:34:29.450144 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:34:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:34:36.411979 3798194 memory.go:184] no items to output this cycle I1121 18:34:36.412137 3798194 cpu.go:275] no items to output this cycle E1121 18:34:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:34:46.411980 3798194 memory.go:184] no items to output this cycle I1121 18:34:46.412274 3798194 cpu.go:275] no items to output this cycle E1121 18:34:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:34:56.413267 3798194 memory.go:191] Add success. W1121 18:34:56.413299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:34:56.413310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:34:56.413314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:34:56.417577 3798194 cpu.go:282] Add success. I1121 18:34:56.441091 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:34:56.441664 3798194 net.go:642] Add success. I1121 18:34:56.443212 3798194 net.go:778] primary dev: ETH0 I1121 18:34:56.443226 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:34:56.443238 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:34:57.450274 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:34:57.450442 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:34:57.450497 3798194 disk_worker.go:723] disk space is not compliant W1121 18:34:57.450500 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:34:57.458532 3798194 disk_worker.go:509] system disk:vda1 I1121 18:34:57.458571 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:34:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:34:58.927442 3798194 disk_info.go:125] begin check local disk info of client I1121 18:34:58.931072 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:34:58.931081 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047c3c0 0xc00047c400] I1121 18:34:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:34:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:34:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:34:59.471700 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:35:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:35:06.413255 3798194 memory.go:184] no items to output this cycle I1121 18:35:06.413397 3798194 cpu.go:275] no items to output this cycle E1121 18:35:16.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:35:16.412884 3798194 memory.go:184] no items to output this cycle I1121 18:35:16.413034 3798194 cpu.go:275] no items to output this cycle E1121 18:35:26.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:35:26.413273 3798194 memory.go:191] Add success. I1121 18:35:26.417592 3798194 cpu.go:282] Add success. I1121 18:35:26.441236 3798194 net.go:642] Add success. I1121 18:35:26.442853 3798194 net.go:778] primary dev: ETH0 I1121 18:35:26.442868 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:35:26.442882 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:35:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:35:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:35:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:35:36.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:35:36.411986 3798194 memory.go:184] no items to output this cycle I1121 18:35:36.412145 3798194 cpu.go:275] no items to output this cycle E1121 18:35:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:35:46.411975 3798194 memory.go:184] no items to output this cycle I1121 18:35:46.412137 3798194 cpu.go:275] no items to output this cycle E1121 18:35:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:35:56.412276 3798194 memory.go:191] Add success. W1121 18:35:56.412304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:35:56.412316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:35:56.412319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:35:56.417099 3798194 cpu.go:282] Add success. I1121 18:35:56.441407 3798194 event_worker.go:152] Polling the log file for events... I1121 18:35:56.444432 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:35:56.444901 3798194 net.go:642] Add success. I1121 18:35:56.446610 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795255556, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 18:35:56.446722 3798194 net.go:778] primary dev: ETH0 I1121 18:35:56.446736 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:35:56.446749 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:35:56.455331 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 18:35:57.450678 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 18:35:57.450692 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:35:57.450895 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:35:57.450904 3798194 disk_worker.go:723] disk space is not compliant W1121 18:35:57.450907 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:35:57.459140 3798194 disk_worker.go:509] system disk:vda1 I1121 18:35:57.459195 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 18:35:58.447496 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 18:35:58.447509 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:35:58.932146 3798194 disk_info.go:125] begin check local disk info of client I1121 18:35:58.935786 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:35:58.935795 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004ab280 0xc0004ab2c0] E1121 18:35:59.448999 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 18:35:59.450078 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:35:59.450141 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:35:59.450161 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:35:59.471686 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:36:06.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:36:06.412889 3798194 memory.go:184] no items to output this cycle I1121 18:36:06.413036 3798194 cpu.go:275] no items to output this cycle E1121 18:36:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:36:16.413258 3798194 memory.go:184] no items to output this cycle I1121 18:36:16.413402 3798194 cpu.go:275] no items to output this cycle I1121 18:36:17.536273 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:36:17.536281 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:36:26.411220 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:36:26.411253 3798194 memory.go:191] Add success. I1121 18:36:26.411611 3798194 cpu.go:282] Add success. I1121 18:36:26.442209 3798194 net.go:642] Add success. I1121 18:36:26.446570 3798194 net.go:778] primary dev: ETH0 I1121 18:36:26.446587 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:36:26.446603 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:36:29.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:36:29.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:36:29.450151 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:36:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:36:36.412259 3798194 memory.go:184] no items to output this cycle I1121 18:36:36.412402 3798194 cpu.go:275] no items to output this cycle E1121 18:36:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:36:46.411979 3798194 memory.go:184] no items to output this cycle I1121 18:36:46.412141 3798194 cpu.go:275] no items to output this cycle E1121 18:36:56.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:36:56.412899 3798194 memory.go:191] Add success. W1121 18:36:56.412926 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:36:56.412937 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:36:56.412940 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:36:56.417311 3798194 cpu.go:282] Add success. I1121 18:36:56.441717 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:36:56.442303 3798194 net.go:642] Add success. I1121 18:36:56.445757 3798194 net.go:778] primary dev: ETH0 I1121 18:36:56.445771 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:36:56.445783 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:36:57.450666 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:36:57.450862 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:36:57.450873 3798194 disk_worker.go:723] disk space is not compliant W1121 18:36:57.450876 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:36:57.458959 3798194 disk_worker.go:509] system disk:vda1 I1121 18:36:57.459002 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:36:58.446024 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:36:58.935904 3798194 disk_info.go:125] begin check local disk info of client I1121 18:36:58.939432 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:36:58.939441 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e6280 0xc0003e62c0] I1121 18:36:59.450091 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:36:59.450171 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:36:59.450197 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:36:59.471715 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:37:06.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:37:06.412887 3798194 memory.go:184] no items to output this cycle I1121 18:37:06.413031 3798194 cpu.go:275] no items to output this cycle E1121 18:37:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:37:16.411986 3798194 memory.go:184] no items to output this cycle I1121 18:37:16.412146 3798194 cpu.go:275] no items to output this cycle E1121 18:37:26.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:37:26.412259 3798194 memory.go:191] Add success. I1121 18:37:26.416629 3798194 cpu.go:282] Add success. I1121 18:37:26.442176 3798194 net.go:642] Add success. I1121 18:37:26.443775 3798194 net.go:778] primary dev: ETH0 I1121 18:37:26.443790 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:37:26.443805 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:37:29.450409 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:37:29.450501 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:37:29.450530 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:37:36.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:37:36.411985 3798194 memory.go:184] no items to output this cycle I1121 18:37:36.412145 3798194 cpu.go:275] no items to output this cycle E1121 18:37:46.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:37:46.412265 3798194 memory.go:184] no items to output this cycle I1121 18:37:46.412412 3798194 cpu.go:275] no items to output this cycle E1121 18:37:56.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:37:56.411960 3798194 memory.go:191] Add success. W1121 18:37:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:37:56.412006 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:37:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:37:56.412358 3798194 cpu.go:282] Add success. I1121 18:37:56.441836 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:37:56.442360 3798194 net.go:642] Add success. I1121 18:37:56.446817 3798194 net.go:778] primary dev: ETH0 I1121 18:37:56.446833 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:37:56.446846 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:37:57.450367 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:37:57.450577 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:37:57.450588 3798194 disk_worker.go:723] disk space is not compliant W1121 18:37:57.450591 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:37:57.458651 3798194 disk_worker.go:509] system disk:vda1 I1121 18:37:57.458707 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:37:58.446372 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:37:58.939543 3798194 disk_info.go:125] begin check local disk info of client I1121 18:37:58.943098 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:37:58.943106 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000351c00 0xc000351c40] I1121 18:37:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:37:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:37:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:37:59.471666 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:38:06.412175 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:38:06.412199 3798194 memory.go:184] no items to output this cycle I1121 18:38:06.412353 3798194 cpu.go:275] no items to output this cycle E1121 18:38:16.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:38:16.411993 3798194 memory.go:184] no items to output this cycle I1121 18:38:16.412149 3798194 cpu.go:275] no items to output this cycle E1121 18:38:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:38:26.411957 3798194 memory.go:191] Add success. I1121 18:38:26.412309 3798194 cpu.go:282] Add success. I1121 18:38:26.441743 3798194 net.go:642] Add success. I1121 18:38:26.445950 3798194 net.go:778] primary dev: ETH0 I1121 18:38:26.445967 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:38:26.445981 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:38:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:38:29.450148 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:38:29.450173 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:38:36.412096 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:38:36.412137 3798194 memory.go:184] no items to output this cycle I1121 18:38:36.412280 3798194 cpu.go:275] no items to output this cycle E1121 18:38:46.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:38:46.411989 3798194 memory.go:184] no items to output this cycle I1121 18:38:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 18:38:56.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:38:56.413271 3798194 memory.go:191] Add success. W1121 18:38:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:38:56.413314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:38:56.413317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:38:56.417965 3798194 cpu.go:282] Add success. I1121 18:38:56.441339 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:38:56.442220 3798194 net.go:642] Add success. I1121 18:38:56.448603 3798194 net.go:778] primary dev: ETH0 I1121 18:38:56.448618 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:38:56.448633 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:38:57.447952 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:38:57.448219 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:38:57.448231 3798194 disk_worker.go:723] disk space is not compliant W1121 18:38:57.448234 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:38:57.451056 3798194 disk_worker.go:509] system disk:vda1 I1121 18:38:57.451096 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:38:58.445988 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:38:58.943237 3798194 disk_info.go:125] begin check local disk info of client I1121 18:38:58.946788 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:38:58.946797 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004dd900 0xc0004dd940] I1121 18:38:59.450419 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:38:59.450501 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:38:59.450526 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:38:59.471018 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:39:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:39:06.412258 3798194 memory.go:184] no items to output this cycle I1121 18:39:06.412404 3798194 cpu.go:275] no items to output this cycle E1121 18:39:16.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:39:16.412880 3798194 memory.go:184] no items to output this cycle I1121 18:39:16.413033 3798194 cpu.go:275] no items to output this cycle I1121 18:39:17.536448 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:39:17.536456 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:39:26.411419 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:39:26.411452 3798194 memory.go:191] Add success. I1121 18:39:26.411800 3798194 cpu.go:282] Add success. I1121 18:39:26.441440 3798194 net.go:642] Add success. I1121 18:39:26.445702 3798194 net.go:778] primary dev: ETH0 I1121 18:39:26.445720 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:39:26.445733 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:39:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:39:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:39:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:39:36.412437 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:39:36.412461 3798194 memory.go:184] no items to output this cycle I1121 18:39:36.412605 3798194 cpu.go:275] no items to output this cycle E1121 18:39:46.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:39:46.412252 3798194 memory.go:184] no items to output this cycle I1121 18:39:46.412409 3798194 cpu.go:275] no items to output this cycle E1121 18:39:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:39:56.411957 3798194 memory.go:191] Add success. W1121 18:39:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:39:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:39:56.412006 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:39:56.412359 3798194 cpu.go:282] Add success. I1121 18:39:56.441849 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:39:56.442443 3798194 net.go:642] Add success. I1121 18:39:56.446381 3798194 net.go:778] primary dev: ETH0 I1121 18:39:56.446396 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:39:56.446408 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:39:57.450377 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:39:57.450545 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:39:57.450599 3798194 disk_worker.go:723] disk space is not compliant W1121 18:39:57.450603 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:39:57.463187 3798194 disk_worker.go:509] system disk:vda1 I1121 18:39:57.463232 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:39:58.446391 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:39:58.946894 3798194 disk_info.go:125] begin check local disk info of client I1121 18:39:58.950498 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:39:58.950507 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000418780 0xc0004187c0] I1121 18:39:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:39:59.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:39:59.450128 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:39:59.471663 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:40:06.412764 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:40:06.412787 3798194 memory.go:184] no items to output this cycle I1121 18:40:06.412940 3798194 cpu.go:275] no items to output this cycle E1121 18:40:16.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:40:16.411990 3798194 memory.go:184] no items to output this cycle I1121 18:40:16.412153 3798194 cpu.go:275] no items to output this cycle E1121 18:40:26.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:40:26.412899 3798194 memory.go:191] Add success. I1121 18:40:26.417343 3798194 cpu.go:282] Add success. I1121 18:40:26.441862 3798194 net.go:642] Add success. I1121 18:40:26.443336 3798194 net.go:778] primary dev: ETH0 I1121 18:40:26.443350 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:40:26.443363 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:40:29.450407 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:40:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:40:29.450514 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:40:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:40:36.411977 3798194 memory.go:184] no items to output this cycle I1121 18:40:36.412143 3798194 cpu.go:275] no items to output this cycle E1121 18:40:46.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:40:46.412265 3798194 memory.go:184] no items to output this cycle I1121 18:40:46.412428 3798194 cpu.go:275] no items to output this cycle E1121 18:40:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:40:56.411952 3798194 memory.go:191] Add success. W1121 18:40:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:40:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:40:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:40:56.412366 3798194 cpu.go:282] Add success. I1121 18:40:56.444383 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:40:56.444869 3798194 net.go:642] Add success. I1121 18:40:56.448972 3798194 net.go:778] primary dev: ETH0 I1121 18:40:56.448988 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:40:56.449000 3798194 net.go:659] Add eni metrics success, count: 7 E1121 18:40:57.450840 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 18:40:57.450854 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:40:57.451013 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:40:57.451068 3798194 disk_worker.go:723] disk space is not compliant W1121 18:40:57.451071 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:40:57.459307 3798194 disk_worker.go:509] system disk:vda1 I1121 18:40:57.459347 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 18:40:58.447760 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 18:40:58.447775 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:40:58.952143 3798194 disk_info.go:125] begin check local disk info of client I1121 18:40:58.955651 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:40:58.955664 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00014d400 0xc00014d440] E1121 18:40:59.448618 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 18:40:59.449694 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:40:59.449745 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:40:59.449776 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:40:59.471322 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:41:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:41:06.412256 3798194 memory.go:184] no items to output this cycle I1121 18:41:06.412399 3798194 cpu.go:275] no items to output this cycle E1121 18:41:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:41:16.411980 3798194 memory.go:184] no items to output this cycle I1121 18:41:16.412150 3798194 cpu.go:275] no items to output this cycle E1121 18:41:26.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:41:26.411988 3798194 memory.go:191] Add success. I1121 18:41:26.412314 3798194 cpu.go:282] Add success. I1121 18:41:26.441764 3798194 net.go:642] Add success. I1121 18:41:26.445759 3798194 net.go:778] primary dev: ETH0 I1121 18:41:26.445774 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:41:26.445790 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:41:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:41:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:41:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:41:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:41:36.413255 3798194 memory.go:184] no items to output this cycle I1121 18:41:36.413413 3798194 cpu.go:275] no items to output this cycle E1121 18:41:46.412967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:41:46.413003 3798194 memory.go:184] no items to output this cycle I1121 18:41:46.413303 3798194 cpu.go:275] no items to output this cycle E1121 18:41:56.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:41:56.413263 3798194 memory.go:191] Add success. W1121 18:41:56.413299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:41:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:41:56.413315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:41:56.417851 3798194 cpu.go:282] Add success. I1121 18:41:56.441375 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:41:56.442259 3798194 net.go:642] Add success. I1121 18:41:56.443875 3798194 net.go:778] primary dev: ETH0 I1121 18:41:56.443898 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:41:56.443917 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:41:57.450372 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:41:57.450594 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:41:57.450605 3798194 disk_worker.go:723] disk space is not compliant W1121 18:41:57.450608 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:41:57.458696 3798194 disk_worker.go:509] system disk:vda1 I1121 18:41:57.458738 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:41:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:41:58.955770 3798194 disk_info.go:125] begin check local disk info of client I1121 18:41:58.959309 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:41:58.959317 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000574e40 0xc000574e80] I1121 18:41:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:41:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:41:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:41:59.471669 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:42:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:42:06.412260 3798194 memory.go:184] no items to output this cycle I1121 18:42:06.412407 3798194 cpu.go:275] no items to output this cycle E1121 18:42:16.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:42:16.413260 3798194 memory.go:184] no items to output this cycle I1121 18:42:16.413410 3798194 cpu.go:275] no items to output this cycle I1121 18:42:17.536619 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:42:17.536628 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:42:26.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:42:26.412888 3798194 memory.go:191] Add success. I1121 18:42:26.417378 3798194 cpu.go:282] Add success. I1121 18:42:26.441962 3798194 net.go:642] Add success. I1121 18:42:26.443431 3798194 net.go:778] primary dev: ETH0 I1121 18:42:26.443445 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:42:26.443457 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:42:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:42:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:42:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:42:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:42:36.412250 3798194 memory.go:184] no items to output this cycle I1121 18:42:36.412399 3798194 cpu.go:275] no items to output this cycle E1121 18:42:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:42:46.411976 3798194 memory.go:184] no items to output this cycle I1121 18:42:46.412130 3798194 cpu.go:275] no items to output this cycle E1121 18:42:56.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:42:56.412270 3798194 memory.go:191] Add success. W1121 18:42:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:42:56.412310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:42:56.412314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:42:56.416870 3798194 cpu.go:282] Add success. I1121 18:42:56.441264 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:42:56.441867 3798194 net.go:642] Add success. I1121 18:42:56.443295 3798194 net.go:778] primary dev: ETH0 I1121 18:42:56.443311 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:42:56.443325 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:42:57.450422 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:42:57.450577 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:42:57.450627 3798194 disk_worker.go:723] disk space is not compliant W1121 18:42:57.450630 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:42:57.459542 3798194 disk_worker.go:509] system disk:vda1 I1121 18:42:57.459582 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:42:58.446012 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:42:58.959420 3798194 disk_info.go:125] begin check local disk info of client I1121 18:42:58.962948 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:42:58.962957 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004c1440 0xc0004c1480] I1121 18:42:59.450031 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:42:59.450137 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:42:59.450179 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:42:59.471699 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:43:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:43:06.411980 3798194 memory.go:184] no items to output this cycle I1121 18:43:06.412131 3798194 cpu.go:275] no items to output this cycle E1121 18:43:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:43:16.411985 3798194 memory.go:184] no items to output this cycle I1121 18:43:16.412139 3798194 cpu.go:275] no items to output this cycle E1121 18:43:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:43:26.411952 3798194 memory.go:191] Add success. I1121 18:43:26.412313 3798194 cpu.go:282] Add success. I1121 18:43:26.441751 3798194 net.go:642] Add success. I1121 18:43:26.446041 3798194 net.go:778] primary dev: ETH0 I1121 18:43:26.446057 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:43:26.446072 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:43:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:43:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:43:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:43:36.412766 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:43:36.412787 3798194 memory.go:184] no items to output this cycle I1121 18:43:36.412940 3798194 cpu.go:275] no items to output this cycle E1121 18:43:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:43:46.411982 3798194 memory.go:184] no items to output this cycle I1121 18:43:46.412147 3798194 cpu.go:275] no items to output this cycle E1121 18:43:56.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:43:56.412885 3798194 memory.go:191] Add success. W1121 18:43:56.412912 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:43:56.412923 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:43:56.412926 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:43:56.417486 3798194 cpu.go:282] Add success. I1121 18:43:56.441871 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:43:56.442367 3798194 net.go:642] Add success. I1121 18:43:56.443965 3798194 net.go:778] primary dev: ETH0 I1121 18:43:56.443980 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:43:56.443992 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:43:57.450223 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:43:57.450380 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:43:57.450433 3798194 disk_worker.go:723] disk space is not compliant W1121 18:43:57.450447 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:43:57.458427 3798194 disk_worker.go:509] system disk:vda1 I1121 18:43:57.458481 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:43:58.446199 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:43:58.963053 3798194 disk_info.go:125] begin check local disk info of client I1121 18:43:58.966660 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:43:58.966669 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003bca80 0xc0003bcac0] I1121 18:43:59.450418 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:43:59.450501 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:43:59.450527 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:43:59.471055 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:44:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:44:06.411987 3798194 memory.go:184] no items to output this cycle I1121 18:44:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 18:44:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:44:16.411980 3798194 memory.go:184] no items to output this cycle I1121 18:44:16.412138 3798194 cpu.go:275] no items to output this cycle E1121 18:44:26.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:44:26.412902 3798194 memory.go:191] Add success. I1121 18:44:26.417563 3798194 cpu.go:282] Add success. I1121 18:44:26.442133 3798194 net.go:642] Add success. I1121 18:44:26.443625 3798194 net.go:778] primary dev: ETH0 I1121 18:44:26.443638 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:44:26.443651 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:44:29.450002 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:44:29.450080 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:44:29.450125 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:44:36.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:44:36.411970 3798194 memory.go:184] no items to output this cycle I1121 18:44:36.412140 3798194 cpu.go:275] no items to output this cycle E1121 18:44:46.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:44:46.411985 3798194 memory.go:184] no items to output this cycle I1121 18:44:46.412150 3798194 cpu.go:275] no items to output this cycle E1121 18:44:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:44:56.412264 3798194 memory.go:191] Add success. W1121 18:44:56.412295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:44:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:44:56.412310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:44:56.416814 3798194 cpu.go:282] Add success. I1121 18:44:56.441199 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:44:56.441675 3798194 net.go:642] Add success. I1121 18:44:56.443424 3798194 net.go:778] primary dev: ETH0 I1121 18:44:56.443437 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:44:56.443450 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:44:57.453635 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:44:57.453859 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:44:57.453879 3798194 disk_worker.go:723] disk space is not compliant W1121 18:44:57.453882 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:44:57.463183 3798194 disk_worker.go:509] system disk:vda1 I1121 18:44:57.463223 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:44:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:44:58.966774 3798194 disk_info.go:125] begin check local disk info of client I1121 18:44:58.970678 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:44:58.970687 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000309300 0xc000309340] I1121 18:44:59.450418 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:44:59.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:44:59.450512 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:44:59.471055 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:45:06.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:45:06.412258 3798194 memory.go:184] no items to output this cycle I1121 18:45:06.412409 3798194 cpu.go:275] no items to output this cycle E1121 18:45:16.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:45:16.412258 3798194 memory.go:184] no items to output this cycle I1121 18:45:16.412405 3798194 cpu.go:275] no items to output this cycle I1121 18:45:17.537267 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:45:17.537277 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:45:26.411356 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:45:26.411389 3798194 memory.go:191] Add success. I1121 18:45:26.411702 3798194 cpu.go:282] Add success. I1121 18:45:26.441303 3798194 net.go:642] Add success. I1121 18:45:26.445440 3798194 net.go:778] primary dev: ETH0 I1121 18:45:26.445456 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:45:26.445469 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:45:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:45:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:45:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:45:36.412809 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:45:36.412832 3798194 memory.go:184] no items to output this cycle I1121 18:45:36.412979 3798194 cpu.go:275] no items to output this cycle E1121 18:45:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:45:46.411978 3798194 memory.go:184] no items to output this cycle I1121 18:45:46.412250 3798194 cpu.go:275] no items to output this cycle E1121 18:45:56.412223 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:45:56.412257 3798194 memory.go:191] Add success. W1121 18:45:56.412284 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:45:56.412295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:45:56.412299 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:45:56.416893 3798194 cpu.go:282] Add success. I1121 18:45:56.445317 3798194 event_worker.go:152] Polling the log file for events... I1121 18:45:56.445537 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:45:56.446000 3798194 net.go:642] Add success. I1121 18:45:56.447345 3798194 net.go:778] primary dev: ETH0 I1121 18:45:56.447368 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:45:56.447382 3798194 net.go:659] Add eni metrics success, count: 7 E1121 18:45:57.450650 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 18:45:57.450664 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:45:57.450857 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:45:57.450867 3798194 disk_worker.go:723] disk space is not compliant W1121 18:45:57.450870 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:45:57.458911 3798194 disk_worker.go:509] system disk:vda1 I1121 18:45:57.458952 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 18:45:58.448160 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 18:45:58.448175 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:45:58.970795 3798194 disk_info.go:125] begin check local disk info of client I1121 18:45:58.974357 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:45:58.974368 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e4d80 0xc0002e4dc0] E1121 18:45:59.449058 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 18:45:59.450125 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:45:59.450176 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:45:59.450203 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:45:59.470676 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:46:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:46:06.411978 3798194 memory.go:184] no items to output this cycle I1121 18:46:06.412139 3798194 cpu.go:275] no items to output this cycle E1121 18:46:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:46:16.411971 3798194 memory.go:184] no items to output this cycle I1121 18:46:16.412138 3798194 cpu.go:275] no items to output this cycle E1121 18:46:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:46:26.411953 3798194 memory.go:191] Add success. I1121 18:46:26.412297 3798194 cpu.go:282] Add success. I1121 18:46:26.441748 3798194 net.go:642] Add success. I1121 18:46:26.446011 3798194 net.go:778] primary dev: ETH0 I1121 18:46:26.446027 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:46:26.446039 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:46:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:46:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:46:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:46:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:46:36.413257 3798194 memory.go:184] no items to output this cycle I1121 18:46:36.413398 3798194 cpu.go:275] no items to output this cycle E1121 18:46:46.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:46:46.412253 3798194 memory.go:184] no items to output this cycle I1121 18:46:46.412408 3798194 cpu.go:275] no items to output this cycle E1121 18:46:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:46:56.411951 3798194 memory.go:191] Add success. W1121 18:46:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:46:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:46:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:46:56.412324 3798194 cpu.go:282] Add success. I1121 18:46:56.441806 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:46:56.442326 3798194 net.go:642] Add success. I1121 18:46:56.446721 3798194 net.go:778] primary dev: ETH0 I1121 18:46:56.446736 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:46:56.446750 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:46:57.451227 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:46:57.451385 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:46:57.451438 3798194 disk_worker.go:723] disk space is not compliant W1121 18:46:57.451442 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:46:57.460227 3798194 disk_worker.go:509] system disk:vda1 I1121 18:46:57.460264 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:46:58.446217 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:46:58.974471 3798194 disk_info.go:125] begin check local disk info of client I1121 18:46:58.977994 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:46:58.978002 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004771c0 0xc000477200] I1121 18:46:59.450415 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:46:59.450493 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:46:59.450518 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:46:59.471045 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:47:06.413159 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:47:06.413181 3798194 memory.go:184] no items to output this cycle I1121 18:47:06.413338 3798194 cpu.go:275] no items to output this cycle E1121 18:47:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:47:16.411971 3798194 memory.go:184] no items to output this cycle I1121 18:47:16.412125 3798194 cpu.go:275] no items to output this cycle E1121 18:47:26.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:47:26.412890 3798194 memory.go:191] Add success. I1121 18:47:26.417153 3798194 cpu.go:282] Add success. I1121 18:47:26.441644 3798194 net.go:642] Add success. I1121 18:47:26.443150 3798194 net.go:778] primary dev: ETH0 I1121 18:47:26.443166 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:47:26.443179 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:47:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:47:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:47:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:47:36.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:47:36.412888 3798194 memory.go:184] no items to output this cycle I1121 18:47:36.413030 3798194 cpu.go:275] no items to output this cycle E1121 18:47:46.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:47:46.412883 3798194 memory.go:184] no items to output this cycle I1121 18:47:46.413033 3798194 cpu.go:275] no items to output this cycle E1121 18:47:56.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:47:56.413286 3798194 memory.go:191] Add success. W1121 18:47:56.413319 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:47:56.413330 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:47:56.413333 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:47:56.417626 3798194 cpu.go:282] Add success. I1121 18:47:56.440941 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:47:56.441522 3798194 net.go:642] Add success. I1121 18:47:56.442978 3798194 net.go:778] primary dev: ETH0 I1121 18:47:56.442992 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:47:56.443003 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:47:57.451032 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:47:57.451271 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:47:57.451284 3798194 disk_worker.go:723] disk space is not compliant W1121 18:47:57.451287 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:47:57.460007 3798194 disk_worker.go:509] system disk:vda1 I1121 18:47:57.460049 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:47:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:47:58.978100 3798194 disk_info.go:125] begin check local disk info of client I1121 18:47:58.981897 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:47:58.981906 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f21c0 0xc0003f2200] I1121 18:47:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:47:59.450134 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:47:59.450165 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:47:59.470703 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:48:06.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:48:06.411973 3798194 memory.go:184] no items to output this cycle I1121 18:48:06.412127 3798194 cpu.go:275] no items to output this cycle E1121 18:48:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:48:16.411975 3798194 memory.go:184] no items to output this cycle I1121 18:48:16.412131 3798194 cpu.go:275] no items to output this cycle I1121 18:48:17.537436 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:48:17.537445 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:48:26.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:48:26.412272 3798194 memory.go:191] Add success. I1121 18:48:26.416701 3798194 cpu.go:282] Add success. I1121 18:48:26.441289 3798194 net.go:642] Add success. I1121 18:48:26.442774 3798194 net.go:778] primary dev: ETH0 I1121 18:48:26.442788 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:48:26.442801 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:48:29.450132 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:48:29.450204 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:48:29.450231 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:48:36.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:48:36.412880 3798194 memory.go:184] no items to output this cycle I1121 18:48:36.413023 3798194 cpu.go:275] no items to output this cycle E1121 18:48:46.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:48:46.411982 3798194 memory.go:184] no items to output this cycle I1121 18:48:46.412138 3798194 cpu.go:275] no items to output this cycle E1121 18:48:56.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:48:56.411984 3798194 memory.go:191] Add success. W1121 18:48:56.412015 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:48:56.412027 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:48:56.412030 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:48:56.412362 3798194 cpu.go:282] Add success. I1121 18:48:56.441805 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:48:56.442372 3798194 net.go:642] Add success. I1121 18:48:56.447100 3798194 net.go:778] primary dev: ETH0 I1121 18:48:56.447131 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:48:56.447144 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:48:57.450070 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:48:57.450257 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:48:57.450313 3798194 disk_worker.go:723] disk space is not compliant W1121 18:48:57.450316 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:48:57.458409 3798194 disk_worker.go:509] system disk:vda1 I1121 18:48:57.458452 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:48:58.446486 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:48:58.982027 3798194 disk_info.go:125] begin check local disk info of client I1121 18:48:58.985537 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:48:58.985546 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00043c640 0xc00043c680] I1121 18:48:59.450411 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:48:59.450487 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:48:59.450517 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:48:59.471045 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:49:06.412967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:49:06.412989 3798194 memory.go:184] no items to output this cycle I1121 18:49:06.413160 3798194 cpu.go:275] no items to output this cycle E1121 18:49:16.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:49:16.413257 3798194 memory.go:184] no items to output this cycle I1121 18:49:16.413402 3798194 cpu.go:275] no items to output this cycle E1121 18:49:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:49:26.411947 3798194 memory.go:191] Add success. I1121 18:49:26.412296 3798194 cpu.go:282] Add success. I1121 18:49:26.441762 3798194 net.go:642] Add success. I1121 18:49:26.445722 3798194 net.go:778] primary dev: ETH0 I1121 18:49:26.445736 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:49:26.445751 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:49:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:49:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:49:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:49:36.412776 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:49:36.412806 3798194 memory.go:184] no items to output this cycle I1121 18:49:36.412951 3798194 cpu.go:275] no items to output this cycle E1121 18:49:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:49:46.411977 3798194 memory.go:184] no items to output this cycle I1121 18:49:46.412143 3798194 cpu.go:275] no items to output this cycle E1121 18:49:56.412221 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:49:56.412254 3798194 memory.go:191] Add success. W1121 18:49:56.412287 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:49:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:49:56.412302 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:49:56.416688 3798194 cpu.go:282] Add success. I1121 18:49:56.440958 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:49:56.441551 3798194 net.go:642] Add success. I1121 18:49:56.443096 3798194 net.go:778] primary dev: ETH0 I1121 18:49:56.443123 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:49:56.443139 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:49:57.450343 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:49:57.450518 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:49:57.450572 3798194 disk_worker.go:723] disk space is not compliant W1121 18:49:57.450575 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:49:57.458705 3798194 disk_worker.go:509] system disk:vda1 I1121 18:49:57.458748 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:49:58.445995 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:49:58.987147 3798194 disk_info.go:125] begin check local disk info of client I1121 18:49:58.990701 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:49:58.990712 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071e40 0xc000071e80] I1121 18:49:59.450422 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:49:59.450508 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:49:59.450534 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:49:59.471077 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:50:06.413252 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:50:06.413274 3798194 memory.go:184] no items to output this cycle I1121 18:50:06.413419 3798194 cpu.go:275] no items to output this cycle E1121 18:50:16.411985 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:50:16.412009 3798194 memory.go:184] no items to output this cycle I1121 18:50:16.412173 3798194 cpu.go:275] no items to output this cycle E1121 18:50:26.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:50:26.413276 3798194 memory.go:191] Add success. I1121 18:50:26.418212 3798194 cpu.go:282] Add success. I1121 18:50:26.441741 3798194 net.go:642] Add success. I1121 18:50:26.443272 3798194 net.go:778] primary dev: ETH0 I1121 18:50:26.443288 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:50:26.443300 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:50:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:50:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:50:29.450498 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:50:36.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:50:36.411982 3798194 memory.go:184] no items to output this cycle I1121 18:50:36.412137 3798194 cpu.go:275] no items to output this cycle E1121 18:50:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:50:46.411981 3798194 memory.go:184] no items to output this cycle I1121 18:50:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 18:50:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:50:56.411945 3798194 memory.go:191] Add success. W1121 18:50:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:50:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:50:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:50:56.412353 3798194 cpu.go:282] Add success. I1121 18:50:56.444395 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:50:56.444875 3798194 net.go:642] Add success. I1121 18:50:56.449084 3798194 net.go:778] primary dev: ETH0 I1121 18:50:56.449098 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:50:56.449125 3798194 net.go:659] Add eni metrics success, count: 7 E1121 18:50:57.450759 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 18:50:57.450774 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:50:57.450947 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:50:57.451001 3798194 disk_worker.go:723] disk space is not compliant W1121 18:50:57.451004 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:50:57.459204 3798194 disk_worker.go:509] system disk:vda1 I1121 18:50:57.459245 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 18:50:58.448197 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 18:50:58.448211 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:50:58.992140 3798194 disk_info.go:125] begin check local disk info of client I1121 18:50:58.996056 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:50:58.996065 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00045ccc0 0xc00045cd00] E1121 18:50:59.449017 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 18:50:59.450091 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:50:59.450152 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:50:59.450171 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:50:59.471687 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:51:06.412094 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:51:06.412136 3798194 memory.go:184] no items to output this cycle I1121 18:51:06.412280 3798194 cpu.go:275] no items to output this cycle E1121 18:51:16.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:51:16.412890 3798194 memory.go:184] no items to output this cycle I1121 18:51:16.413036 3798194 cpu.go:275] no items to output this cycle I1121 18:51:17.538525 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:51:17.538533 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:51:26.411447 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:51:26.411490 3798194 memory.go:191] Add success. I1121 18:51:26.411846 3798194 cpu.go:282] Add success. I1121 18:51:26.441473 3798194 net.go:642] Add success. I1121 18:51:26.448657 3798194 net.go:778] primary dev: ETH0 I1121 18:51:26.448686 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:51:26.448707 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:51:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:51:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:51:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:51:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:51:36.413262 3798194 memory.go:184] no items to output this cycle I1121 18:51:36.413415 3798194 cpu.go:275] no items to output this cycle E1121 18:51:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:51:46.411980 3798194 memory.go:184] no items to output this cycle I1121 18:51:46.412134 3798194 cpu.go:275] no items to output this cycle E1121 18:51:56.415849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:51:56.415896 3798194 memory.go:191] Add success. W1121 18:51:56.415932 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:51:56.415948 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:51:56.415953 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:51:56.437491 3798194 cpu.go:282] Add success. I1121 18:51:56.441075 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:51:56.441926 3798194 net.go:642] Add success. I1121 18:51:56.443592 3798194 net.go:778] primary dev: ETH0 I1121 18:51:56.443614 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:51:56.443637 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:51:57.445281 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:51:57.445501 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:51:57.445571 3798194 disk_worker.go:723] disk space is not compliant W1121 18:51:57.445574 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:51:57.448051 3798194 disk_worker.go:509] system disk:vda1 I1121 18:51:57.448093 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:51:58.445983 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:51:58.996170 3798194 disk_info.go:125] begin check local disk info of client I1121 18:51:58.999780 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:51:58.999789 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a4200 0xc0003a4240] I1121 18:51:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:51:59.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:51:59.450151 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:51:59.471670 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:52:06.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:52:06.411974 3798194 memory.go:184] no items to output this cycle I1121 18:52:06.412143 3798194 cpu.go:275] no items to output this cycle E1121 18:52:16.412253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:52:16.412274 3798194 memory.go:184] no items to output this cycle I1121 18:52:16.412429 3798194 cpu.go:275] no items to output this cycle E1121 18:52:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:52:26.411956 3798194 memory.go:191] Add success. I1121 18:52:26.412315 3798194 cpu.go:282] Add success. I1121 18:52:26.442254 3798194 net.go:642] Add success. I1121 18:52:26.446470 3798194 net.go:778] primary dev: ETH0 I1121 18:52:26.446485 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:52:26.446497 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:52:29.450315 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:52:29.450398 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:52:29.450424 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:52:36.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:52:36.413253 3798194 memory.go:184] no items to output this cycle I1121 18:52:36.413405 3798194 cpu.go:275] no items to output this cycle E1121 18:52:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:52:46.413265 3798194 memory.go:184] no items to output this cycle I1121 18:52:46.413509 3798194 cpu.go:275] no items to output this cycle E1121 18:52:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:52:56.411954 3798194 memory.go:191] Add success. W1121 18:52:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:52:56.412001 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:52:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:52:56.412365 3798194 cpu.go:282] Add success. I1121 18:52:56.441163 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:52:56.441634 3798194 net.go:642] Add success. I1121 18:52:56.445916 3798194 net.go:778] primary dev: ETH0 I1121 18:52:56.445933 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:52:56.445946 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:52:57.450397 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:52:57.450603 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:52:57.450614 3798194 disk_worker.go:723] disk space is not compliant W1121 18:52:57.450617 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:52:57.458727 3798194 disk_worker.go:509] system disk:vda1 I1121 18:52:57.458768 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:52:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:52:58.999899 3798194 disk_info.go:125] begin check local disk info of client I1121 18:52:59.003384 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:52:59.003393 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047c380 0xc00047c3c0] I1121 18:52:59.450412 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:52:59.450501 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:52:59.450529 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:52:59.471121 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:53:06.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:53:06.411977 3798194 memory.go:184] no items to output this cycle I1121 18:53:06.412131 3798194 cpu.go:275] no items to output this cycle E1121 18:53:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:53:16.411987 3798194 memory.go:184] no items to output this cycle I1121 18:53:16.412140 3798194 cpu.go:275] no items to output this cycle E1121 18:53:26.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:53:26.412260 3798194 memory.go:191] Add success. I1121 18:53:26.417443 3798194 cpu.go:282] Add success. I1121 18:53:26.441962 3798194 net.go:642] Add success. I1121 18:53:26.443615 3798194 net.go:778] primary dev: ETH0 I1121 18:53:26.443630 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:53:26.443642 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:53:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:53:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:53:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:53:36.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:53:36.412879 3798194 memory.go:184] no items to output this cycle I1121 18:53:36.413019 3798194 cpu.go:275] no items to output this cycle E1121 18:53:46.413763 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:53:46.413786 3798194 memory.go:184] no items to output this cycle I1121 18:53:46.413928 3798194 cpu.go:275] no items to output this cycle E1121 18:53:56.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:53:56.412276 3798194 memory.go:191] Add success. W1121 18:53:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:53:56.412318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:53:56.412322 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:53:56.416858 3798194 cpu.go:282] Add success. I1121 18:53:56.441273 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:53:56.441792 3798194 net.go:642] Add success. I1121 18:53:56.443409 3798194 net.go:778] primary dev: ETH0 I1121 18:53:56.443424 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:53:56.443437 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:53:57.455178 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:53:57.455389 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:53:57.455400 3798194 disk_worker.go:723] disk space is not compliant W1121 18:53:57.455403 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:53:57.463773 3798194 disk_worker.go:509] system disk:vda1 I1121 18:53:57.463816 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:53:58.446500 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:53:59.003501 3798194 disk_info.go:125] begin check local disk info of client I1121 18:53:59.007087 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:53:59.007096 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00048a400 0xc00048a440] I1121 18:53:59.450030 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:53:59.450122 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:53:59.450150 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:53:59.471689 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:54:06.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:54:06.412265 3798194 memory.go:184] no items to output this cycle I1121 18:54:06.412411 3798194 cpu.go:275] no items to output this cycle E1121 18:54:16.413679 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:54:16.413709 3798194 memory.go:184] no items to output this cycle I1121 18:54:16.413859 3798194 cpu.go:275] no items to output this cycle I1121 18:54:17.538703 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:54:17.538713 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:54:26.412717 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:54:26.412760 3798194 memory.go:191] Add success. I1121 18:54:26.417327 3798194 cpu.go:282] Add success. I1121 18:54:26.441738 3798194 net.go:642] Add success. I1121 18:54:26.443244 3798194 net.go:778] primary dev: ETH0 I1121 18:54:26.443258 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:54:26.443271 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:54:29.449991 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:54:29.450073 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:54:29.450104 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:54:36.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:54:36.412267 3798194 memory.go:184] no items to output this cycle I1121 18:54:36.412409 3798194 cpu.go:275] no items to output this cycle E1121 18:54:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:54:46.413259 3798194 memory.go:184] no items to output this cycle I1121 18:54:46.413406 3798194 cpu.go:275] no items to output this cycle E1121 18:54:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:54:56.412265 3798194 memory.go:191] Add success. W1121 18:54:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:54:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:54:56.412313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:54:56.418303 3798194 cpu.go:282] Add success. I1121 18:54:56.441787 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:54:56.442359 3798194 net.go:642] Add success. I1121 18:54:56.443939 3798194 net.go:778] primary dev: ETH0 I1121 18:54:56.443955 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:54:56.443975 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:54:57.448942 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:54:57.449209 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:54:57.449221 3798194 disk_worker.go:723] disk space is not compliant W1121 18:54:57.449224 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:54:57.451034 3798194 disk_worker.go:509] system disk:vda1 I1121 18:54:57.451074 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:54:58.446373 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:54:59.007190 3798194 disk_info.go:125] begin check local disk info of client I1121 18:54:59.010716 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:54:59.010725 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000389740 0xc000389780] I1121 18:54:59.450422 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:54:59.450489 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:54:59.450515 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:54:59.471038 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:55:06.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:55:06.413258 3798194 memory.go:184] no items to output this cycle I1121 18:55:06.413408 3798194 cpu.go:275] no items to output this cycle E1121 18:55:16.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:55:16.413264 3798194 memory.go:184] no items to output this cycle I1121 18:55:16.413414 3798194 cpu.go:275] no items to output this cycle E1121 18:55:26.412883 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:55:26.412918 3798194 memory.go:191] Add success. I1121 18:55:26.417228 3798194 cpu.go:282] Add success. I1121 18:55:26.441733 3798194 net.go:642] Add success. I1121 18:55:26.443264 3798194 net.go:778] primary dev: ETH0 I1121 18:55:26.443278 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:55:26.443405 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:55:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:55:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:55:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:55:36.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:55:36.413257 3798194 memory.go:184] no items to output this cycle I1121 18:55:36.413407 3798194 cpu.go:275] no items to output this cycle E1121 18:55:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:55:46.413262 3798194 memory.go:184] no items to output this cycle I1121 18:55:46.413411 3798194 cpu.go:275] no items to output this cycle E1121 18:55:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:55:56.412264 3798194 memory.go:191] Add success. W1121 18:55:56.412296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:55:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:55:56.412312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:55:56.416969 3798194 cpu.go:282] Add success. I1121 18:55:56.443749 3798194 event_worker.go:152] Polling the log file for events... I1121 18:55:56.443968 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:55:56.444559 3798194 net.go:642] Add success. I1121 18:55:56.446063 3798194 net.go:778] primary dev: ETH0 I1121 18:55:56.446077 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:55:56.446089 3798194 net.go:659] Add eni metrics success, count: 7 E1121 18:55:57.450349 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 18:55:57.450365 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:55:57.450569 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:55:57.450578 3798194 disk_worker.go:723] disk space is not compliant W1121 18:55:57.450581 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:55:57.458658 3798194 disk_worker.go:509] system disk:vda1 I1121 18:55:57.458699 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 18:55:58.448151 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 18:55:58.448165 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:55:59.010830 3798194 disk_info.go:125] begin check local disk info of client I1121 18:55:59.014400 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:55:59.014408 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c7540 0xc0003c7580] E1121 18:55:59.448616 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 18:55:59.449691 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:55:59.449740 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:55:59.449758 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:55:59.471268 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:56:06.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:56:06.411975 3798194 memory.go:184] no items to output this cycle I1121 18:56:06.412132 3798194 cpu.go:275] no items to output this cycle E1121 18:56:16.413249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:56:16.413273 3798194 memory.go:184] no items to output this cycle I1121 18:56:16.413424 3798194 cpu.go:275] no items to output this cycle E1121 18:56:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:56:26.411958 3798194 memory.go:191] Add success. I1121 18:56:26.412309 3798194 cpu.go:282] Add success. I1121 18:56:26.442006 3798194 net.go:642] Add success. I1121 18:56:26.446075 3798194 net.go:778] primary dev: ETH0 I1121 18:56:26.446090 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:56:26.446102 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:56:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:56:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:56:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:56:36.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:56:36.412255 3798194 memory.go:184] no items to output this cycle I1121 18:56:36.412399 3798194 cpu.go:275] no items to output this cycle E1121 18:56:46.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:56:46.412269 3798194 memory.go:184] no items to output this cycle I1121 18:56:46.412429 3798194 cpu.go:275] no items to output this cycle E1121 18:56:56.412865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:56:56.412899 3798194 memory.go:191] Add success. W1121 18:56:56.412929 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:56:56.412941 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:56:56.412944 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:56:56.417299 3798194 cpu.go:282] Add success. I1121 18:56:56.441693 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:56:56.442443 3798194 net.go:642] Add success. I1121 18:56:56.444274 3798194 net.go:778] primary dev: ETH0 I1121 18:56:56.444291 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:56:56.444310 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:56:57.452335 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:56:57.452500 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:56:57.452557 3798194 disk_worker.go:723] disk space is not compliant W1121 18:56:57.452560 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:56:57.461407 3798194 disk_worker.go:509] system disk:vda1 I1121 18:56:57.461450 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:56:58.446011 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:56:59.014513 3798194 disk_info.go:125] begin check local disk info of client I1121 18:56:59.017963 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:56:59.017971 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004cd040 0xc0004cd080] I1121 18:56:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:56:59.450082 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:56:59.450107 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:56:59.471620 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:57:06.412365 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:57:06.412387 3798194 memory.go:184] no items to output this cycle I1121 18:57:06.412528 3798194 cpu.go:275] no items to output this cycle E1121 18:57:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:57:16.411985 3798194 memory.go:184] no items to output this cycle I1121 18:57:16.412141 3798194 cpu.go:275] no items to output this cycle I1121 18:57:17.538876 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 18:57:17.538886 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 18:57:26.411278 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:57:26.411313 3798194 memory.go:191] Add success. I1121 18:57:26.411667 3798194 cpu.go:282] Add success. I1121 18:57:26.441277 3798194 net.go:642] Add success. I1121 18:57:26.445060 3798194 net.go:778] primary dev: ETH0 I1121 18:57:26.445076 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:57:26.445091 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:57:29.449943 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:57:29.450023 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:57:29.450051 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:57:36.412315 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:57:36.412338 3798194 memory.go:184] no items to output this cycle I1121 18:57:36.412489 3798194 cpu.go:275] no items to output this cycle E1121 18:57:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:57:46.413256 3798194 memory.go:184] no items to output this cycle I1121 18:57:46.413400 3798194 cpu.go:275] no items to output this cycle E1121 18:57:56.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:57:56.412894 3798194 memory.go:191] Add success. W1121 18:57:56.412929 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:57:56.412941 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:57:56.412944 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:57:56.417315 3798194 cpu.go:282] Add success. I1121 18:57:56.441698 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:57:56.442181 3798194 net.go:642] Add success. I1121 18:57:56.443862 3798194 net.go:778] primary dev: ETH0 I1121 18:57:56.443877 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:57:56.443889 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:57:57.450880 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:57:57.451096 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:57:57.451107 3798194 disk_worker.go:723] disk space is not compliant W1121 18:57:57.451128 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:57:57.464156 3798194 disk_worker.go:509] system disk:vda1 I1121 18:57:57.464207 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:57:58.445984 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:57:59.018065 3798194 disk_info.go:125] begin check local disk info of client I1121 18:57:59.021636 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:57:59.021646 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d5e80 0xc0003d5ec0] I1121 18:57:59.449964 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:57:59.450052 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:57:59.450077 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:57:59.471633 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:58:06.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:58:06.411993 3798194 memory.go:184] no items to output this cycle I1121 18:58:06.412148 3798194 cpu.go:275] no items to output this cycle E1121 18:58:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:58:16.413258 3798194 memory.go:184] no items to output this cycle I1121 18:58:16.413401 3798194 cpu.go:275] no items to output this cycle E1121 18:58:26.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:58:26.411963 3798194 memory.go:191] Add success. I1121 18:58:26.412331 3798194 cpu.go:282] Add success. I1121 18:58:26.441923 3798194 net.go:642] Add success. I1121 18:58:26.446793 3798194 net.go:778] primary dev: ETH0 I1121 18:58:26.446807 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:58:26.446822 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:58:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:58:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:58:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:58:36.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:58:36.412896 3798194 memory.go:184] no items to output this cycle I1121 18:58:36.413039 3798194 cpu.go:275] no items to output this cycle E1121 18:58:46.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:58:46.411991 3798194 memory.go:184] no items to output this cycle I1121 18:58:46.412153 3798194 cpu.go:275] no items to output this cycle E1121 18:58:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:58:56.411950 3798194 memory.go:191] Add success. W1121 18:58:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:58:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:58:56.411996 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:58:56.412336 3798194 cpu.go:282] Add success. I1121 18:58:56.441811 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:58:56.442398 3798194 net.go:642] Add success. I1121 18:58:56.446494 3798194 net.go:778] primary dev: ETH0 I1121 18:58:56.446509 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:58:56.446520 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:58:57.451015 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:58:57.451247 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:58:57.451260 3798194 disk_worker.go:723] disk space is not compliant W1121 18:58:57.451263 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:58:57.461698 3798194 disk_worker.go:509] system disk:vda1 I1121 18:58:57.461740 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:58:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:58:59.021750 3798194 disk_info.go:125] begin check local disk info of client I1121 18:58:59.025614 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:58:59.025622 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000313300 0xc000313340] I1121 18:58:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:58:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:58:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:58:59.471655 3798194 disk_local_worker.go:436] Get disk info: [] E1121 18:59:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:59:06.412258 3798194 memory.go:184] no items to output this cycle I1121 18:59:06.412396 3798194 cpu.go:275] no items to output this cycle E1121 18:59:16.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:59:16.412250 3798194 memory.go:184] no items to output this cycle I1121 18:59:16.412394 3798194 cpu.go:275] no items to output this cycle E1121 18:59:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:59:26.411956 3798194 memory.go:191] Add success. I1121 18:59:26.412306 3798194 cpu.go:282] Add success. I1121 18:59:26.441789 3798194 net.go:642] Add success. I1121 18:59:26.445883 3798194 net.go:778] primary dev: ETH0 I1121 18:59:26.445898 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:59:26.445921 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:59:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:59:29.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:59:29.450122 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 18:59:36.412763 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:59:36.412785 3798194 memory.go:184] no items to output this cycle I1121 18:59:36.412936 3798194 cpu.go:275] no items to output this cycle E1121 18:59:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:59:46.411978 3798194 memory.go:184] no items to output this cycle I1121 18:59:46.412141 3798194 cpu.go:275] no items to output this cycle E1121 18:59:56.412298 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 18:59:56.412353 3798194 memory.go:191] Add success. W1121 18:59:56.412393 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 18:59:56.412412 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 18:59:56.412417 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 18:59:56.417651 3798194 cpu.go:282] Add success. I1121 18:59:56.441506 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 18:59:56.442431 3798194 net.go:642] Add success. I1121 18:59:56.444170 3798194 net.go:778] primary dev: ETH0 I1121 18:59:56.444192 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 18:59:56.444210 3798194 net.go:659] Add eni metrics success, count: 7 I1121 18:59:57.446527 3798194 custom_config.go:106] query custom config with name: gpu W1121 18:59:57.446726 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 18:59:57.446787 3798194 disk_worker.go:723] disk space is not compliant W1121 18:59:57.446791 3798194 disk_worker.go:743] disk inode is not compliant I1121 18:59:57.449532 3798194 disk_worker.go:509] system disk:vda1 I1121 18:59:57.449574 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 18:59:58.446373 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 18:59:59.025720 3798194 disk_info.go:125] begin check local disk info of client I1121 18:59:59.029637 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 18:59:59.029646 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0c40 0xc0002e0c80] I1121 18:59:59.450412 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 18:59:59.450490 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 18:59:59.450517 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 18:59:59.471019 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:00:06.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:00:06.413257 3798194 memory.go:184] no items to output this cycle I1121 19:00:06.413401 3798194 cpu.go:275] no items to output this cycle E1121 19:00:16.413253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:00:16.413280 3798194 memory.go:184] no items to output this cycle I1121 19:00:16.413433 3798194 cpu.go:275] no items to output this cycle I1121 19:00:17.539926 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:00:17.539935 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:00:26.411367 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:00:26.411401 3798194 memory.go:191] Add success. I1121 19:00:26.411738 3798194 cpu.go:282] Add success. I1121 19:00:26.441291 3798194 net.go:642] Add success. I1121 19:00:26.445329 3798194 net.go:778] primary dev: ETH0 I1121 19:00:26.445354 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:00:26.445366 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:00:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:00:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:00:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:00:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:00:36.413263 3798194 memory.go:184] no items to output this cycle I1121 19:00:36.413407 3798194 cpu.go:275] no items to output this cycle E1121 19:00:46.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:00:46.413265 3798194 memory.go:184] no items to output this cycle I1121 19:00:46.413411 3798194 cpu.go:275] no items to output this cycle E1121 19:00:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:00:56.411946 3798194 memory.go:191] Add success. W1121 19:00:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:00:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:00:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:00:56.412339 3798194 cpu.go:282] Add success. I1121 19:00:56.445063 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:00:56.445594 3798194 net.go:642] Add success. I1121 19:00:56.449477 3798194 net.go:778] primary dev: ETH0 I1121 19:00:56.449493 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:00:56.449513 3798194 net.go:659] Add eni metrics success, count: 7 E1121 19:00:57.450643 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 19:00:57.450657 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:00:57.450814 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:00:57.450867 3798194 disk_worker.go:723] disk space is not compliant W1121 19:00:57.450870 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:00:57.459352 3798194 disk_worker.go:509] system disk:vda1 I1121 19:00:57.459397 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 19:00:58.449207 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 19:00:58.449222 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:00:59.029748 3798194 disk_info.go:125] begin check local disk info of client I1121 19:00:59.033585 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:00:59.033594 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004ae040 0xc0004ae0c0] E1121 19:00:59.448627 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 19:00:59.449704 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:00:59.449758 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:00:59.449778 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:00:59.471366 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:01:06.412773 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:01:06.412797 3798194 memory.go:184] no items to output this cycle I1121 19:01:06.412949 3798194 cpu.go:275] no items to output this cycle E1121 19:01:16.420245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:01:16.420269 3798194 memory.go:184] no items to output this cycle I1121 19:01:16.420434 3798194 cpu.go:275] no items to output this cycle E1121 19:01:26.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:01:26.412269 3798194 memory.go:191] Add success. I1121 19:01:26.416543 3798194 cpu.go:282] Add success. I1121 19:01:26.441253 3798194 net.go:642] Add success. I1121 19:01:26.442707 3798194 net.go:778] primary dev: ETH0 I1121 19:01:26.442722 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:01:26.442735 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:01:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:01:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:01:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:01:36.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:01:36.413262 3798194 memory.go:184] no items to output this cycle I1121 19:01:36.413407 3798194 cpu.go:275] no items to output this cycle E1121 19:01:46.411517 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:01:46.411539 3798194 memory.go:184] no items to output this cycle I1121 19:01:46.411697 3798194 cpu.go:275] no items to output this cycle E1121 19:01:56.412447 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:01:56.412482 3798194 memory.go:191] Add success. W1121 19:01:56.412510 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:01:56.412522 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:01:56.412526 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:01:56.416690 3798194 cpu.go:282] Add success. I1121 19:01:56.441090 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:01:56.441586 3798194 net.go:642] Add success. I1121 19:01:56.443245 3798194 net.go:778] primary dev: ETH0 I1121 19:01:56.443263 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:01:56.443274 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:01:57.450618 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:01:57.450775 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:01:57.450830 3798194 disk_worker.go:723] disk space is not compliant W1121 19:01:57.450833 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:01:57.459265 3798194 disk_worker.go:509] system disk:vda1 I1121 19:01:57.459306 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:01:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:01:59.033689 3798194 disk_info.go:125] begin check local disk info of client I1121 19:01:59.037554 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:01:59.037563 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000397500 0xc000397540] I1121 19:01:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:01:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:01:59.450133 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:01:59.471653 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:02:06.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:02:06.412884 3798194 memory.go:184] no items to output this cycle I1121 19:02:06.413035 3798194 cpu.go:275] no items to output this cycle E1121 19:02:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:02:16.411980 3798194 memory.go:184] no items to output this cycle I1121 19:02:16.412134 3798194 cpu.go:275] no items to output this cycle E1121 19:02:26.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:02:26.411947 3798194 memory.go:191] Add success. I1121 19:02:26.412314 3798194 cpu.go:282] Add success. I1121 19:02:26.441930 3798194 net.go:642] Add success. I1121 19:02:26.446156 3798194 net.go:778] primary dev: ETH0 I1121 19:02:26.446172 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:02:26.446186 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:02:29.449923 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:02:29.449998 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:02:29.450034 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:02:36.413249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:02:36.413275 3798194 memory.go:184] no items to output this cycle I1121 19:02:36.413618 3798194 cpu.go:275] no items to output this cycle E1121 19:02:46.412767 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:02:46.412790 3798194 memory.go:184] no items to output this cycle I1121 19:02:46.412942 3798194 cpu.go:275] no items to output this cycle E1121 19:02:56.411241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:02:56.411277 3798194 memory.go:191] Add success. W1121 19:02:56.411311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:02:56.411323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:02:56.411326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:02:56.424195 3798194 cpu.go:282] Add success. I1121 19:02:56.441895 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:02:56.442395 3798194 net.go:642] Add success. I1121 19:02:56.443997 3798194 net.go:778] primary dev: ETH0 I1121 19:02:56.444011 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:02:56.444024 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:02:57.451722 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:02:57.451890 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:02:57.451942 3798194 disk_worker.go:723] disk space is not compliant W1121 19:02:57.451945 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:02:57.460426 3798194 disk_worker.go:509] system disk:vda1 I1121 19:02:57.460470 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:02:58.445968 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:02:59.037665 3798194 disk_info.go:125] begin check local disk info of client I1121 19:02:59.041495 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:02:59.041505 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ecc80 0xc0003eccc0] I1121 19:02:59.450422 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:02:59.450493 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:02:59.450519 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:02:59.471027 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:03:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:03:06.411981 3798194 memory.go:184] no items to output this cycle I1121 19:03:06.412136 3798194 cpu.go:275] no items to output this cycle E1121 19:03:16.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:03:16.412258 3798194 memory.go:184] no items to output this cycle I1121 19:03:16.412404 3798194 cpu.go:275] no items to output this cycle I1121 19:03:17.540918 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:03:17.540926 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:03:26.411317 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:03:26.411350 3798194 memory.go:191] Add success. I1121 19:03:26.411682 3798194 cpu.go:282] Add success. I1121 19:03:26.442309 3798194 net.go:642] Add success. I1121 19:03:26.446609 3798194 net.go:778] primary dev: ETH0 I1121 19:03:26.446625 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:03:26.446649 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:03:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:03:29.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:03:29.450122 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:03:36.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:03:36.412264 3798194 memory.go:184] no items to output this cycle I1121 19:03:36.412409 3798194 cpu.go:275] no items to output this cycle E1121 19:03:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:03:46.413260 3798194 memory.go:184] no items to output this cycle I1121 19:03:46.413405 3798194 cpu.go:275] no items to output this cycle E1121 19:03:56.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:03:56.412265 3798194 memory.go:191] Add success. W1121 19:03:56.412295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:03:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:03:56.412309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:03:56.416535 3798194 cpu.go:282] Add success. I1121 19:03:56.441084 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:03:56.441564 3798194 net.go:642] Add success. I1121 19:03:56.443090 3798194 net.go:778] primary dev: ETH0 I1121 19:03:56.443105 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:03:56.443127 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:03:57.450462 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:03:57.450659 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:03:57.450671 3798194 disk_worker.go:723] disk space is not compliant W1121 19:03:57.450674 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:03:57.459892 3798194 disk_worker.go:509] system disk:vda1 I1121 19:03:57.459933 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:03:58.446019 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:03:59.041607 3798194 disk_info.go:125] begin check local disk info of client I1121 19:03:59.045449 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:03:59.045459 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000493800 0xc000493840] I1121 19:03:59.450408 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:03:59.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:03:59.450511 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:03:59.471036 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:04:06.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:04:06.411981 3798194 memory.go:184] no items to output this cycle I1121 19:04:06.412134 3798194 cpu.go:275] no items to output this cycle E1121 19:04:16.412256 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:04:16.412281 3798194 memory.go:184] no items to output this cycle I1121 19:04:16.412433 3798194 cpu.go:275] no items to output this cycle E1121 19:04:26.412846 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:04:26.412884 3798194 memory.go:191] Add success. I1121 19:04:26.417899 3798194 cpu.go:282] Add success. I1121 19:04:26.441657 3798194 net.go:642] Add success. I1121 19:04:26.443229 3798194 net.go:778] primary dev: ETH0 I1121 19:04:26.443244 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:04:26.443256 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:04:29.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:04:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:04:29.450498 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:04:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:04:36.411984 3798194 memory.go:184] no items to output this cycle I1121 19:04:36.412135 3798194 cpu.go:275] no items to output this cycle E1121 19:04:46.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:04:46.412250 3798194 memory.go:184] no items to output this cycle I1121 19:04:46.412401 3798194 cpu.go:275] no items to output this cycle E1121 19:04:56.412224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:04:56.412256 3798194 memory.go:191] Add success. W1121 19:04:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:04:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:04:56.412312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:04:56.416915 3798194 cpu.go:282] Add success. I1121 19:04:56.441429 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:04:56.441898 3798194 net.go:642] Add success. I1121 19:04:56.443448 3798194 net.go:778] primary dev: ETH0 I1121 19:04:56.443462 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:04:56.443474 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:04:57.450676 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:04:57.450884 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:04:57.450894 3798194 disk_worker.go:723] disk space is not compliant W1121 19:04:57.450898 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:04:57.459257 3798194 disk_worker.go:509] system disk:vda1 I1121 19:04:57.459301 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:04:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:04:59.048414 3798194 disk_info.go:125] begin check local disk info of client I1121 19:04:59.052294 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:04:59.052303 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c6a40 0xc0003c6a80] I1121 19:04:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:04:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:04:59.450140 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:04:59.471638 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:05:06.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:05:06.412876 3798194 memory.go:184] no items to output this cycle I1121 19:05:06.413015 3798194 cpu.go:275] no items to output this cycle E1121 19:05:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:05:16.413255 3798194 memory.go:184] no items to output this cycle I1121 19:05:16.413405 3798194 cpu.go:275] no items to output this cycle E1121 19:05:26.411834 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:05:26.411868 3798194 memory.go:191] Add success. I1121 19:05:26.416241 3798194 cpu.go:282] Add success. I1121 19:05:26.441300 3798194 net.go:642] Add success. I1121 19:05:26.442803 3798194 net.go:778] primary dev: ETH0 I1121 19:05:26.442817 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:05:26.442831 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:05:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:05:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:05:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:05:36.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:05:36.413268 3798194 memory.go:184] no items to output this cycle I1121 19:05:36.413415 3798194 cpu.go:275] no items to output this cycle E1121 19:05:46.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:05:46.413260 3798194 memory.go:184] no items to output this cycle I1121 19:05:46.413415 3798194 cpu.go:275] no items to output this cycle E1121 19:05:56.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:05:56.412884 3798194 memory.go:191] Add success. W1121 19:05:56.412919 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:05:56.412930 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:05:56.412934 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:05:56.417344 3798194 cpu.go:282] Add success. I1121 19:05:56.441497 3798194 event_worker.go:152] Polling the log file for events... I1121 19:05:56.445374 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:05:56.445851 3798194 net.go:642] Add success. I1121 19:05:56.447211 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795257356, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 19:05:56.447324 3798194 net.go:778] primary dev: ETH0 I1121 19:05:56.447340 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:05:56.447351 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:05:56.461863 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 19:05:57.446472 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 19:05:57.446487 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:05:57.446707 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:05:57.446786 3798194 disk_worker.go:723] disk space is not compliant W1121 19:05:57.446789 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:05:57.449624 3798194 disk_worker.go:509] system disk:vda1 I1121 19:05:57.449680 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 19:05:58.448491 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 19:05:58.448507 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:05:59.054146 3798194 disk_info.go:125] begin check local disk info of client I1121 19:05:59.058018 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:05:59.058027 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002feac0 0xc0002feb00] E1121 19:05:59.448940 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 19:05:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:05:59.450068 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:05:59.450093 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:05:59.471569 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:06:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:06:06.411981 3798194 memory.go:184] no items to output this cycle I1121 19:06:06.412145 3798194 cpu.go:275] no items to output this cycle E1121 19:06:16.413249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:06:16.413272 3798194 memory.go:184] no items to output this cycle I1121 19:06:16.413420 3798194 cpu.go:275] no items to output this cycle I1121 19:06:17.541205 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:06:17.541214 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:06:26.411507 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:06:26.411541 3798194 memory.go:191] Add success. I1121 19:06:26.411866 3798194 cpu.go:282] Add success. I1121 19:06:26.441505 3798194 net.go:642] Add success. I1121 19:06:26.445433 3798194 net.go:778] primary dev: ETH0 I1121 19:06:26.445449 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:06:26.445465 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:06:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:06:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:06:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:06:36.412528 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:06:36.412550 3798194 memory.go:184] no items to output this cycle I1121 19:06:36.412704 3798194 cpu.go:275] no items to output this cycle E1121 19:06:46.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:06:46.412873 3798194 memory.go:184] no items to output this cycle I1121 19:06:46.413021 3798194 cpu.go:275] no items to output this cycle E1121 19:06:56.412355 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:06:56.412402 3798194 memory.go:191] Add success. W1121 19:06:56.412434 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:06:56.412525 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:06:56.412529 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:06:56.417525 3798194 cpu.go:282] Add success. I1121 19:06:56.440944 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:06:56.441527 3798194 net.go:642] Add success. I1121 19:06:56.442979 3798194 net.go:778] primary dev: ETH0 I1121 19:06:56.442994 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:06:56.443006 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:06:57.451336 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:06:57.451537 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:06:57.451548 3798194 disk_worker.go:723] disk space is not compliant W1121 19:06:57.451551 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:06:57.459981 3798194 disk_worker.go:509] system disk:vda1 I1121 19:06:57.460024 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:06:58.446519 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:06:59.058155 3798194 disk_info.go:125] begin check local disk info of client I1121 19:06:59.062023 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:06:59.062033 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e380 0xc00030e3c0] I1121 19:06:59.450413 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:06:59.450487 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:06:59.450515 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:06:59.471032 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:07:06.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:07:06.413271 3798194 memory.go:184] no items to output this cycle I1121 19:07:06.413422 3798194 cpu.go:275] no items to output this cycle E1121 19:07:16.412874 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:07:16.412899 3798194 memory.go:184] no items to output this cycle I1121 19:07:16.413054 3798194 cpu.go:275] no items to output this cycle E1121 19:07:26.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:07:26.412271 3798194 memory.go:191] Add success. I1121 19:07:26.416669 3798194 cpu.go:282] Add success. I1121 19:07:26.442211 3798194 net.go:642] Add success. I1121 19:07:26.443800 3798194 net.go:778] primary dev: ETH0 I1121 19:07:26.443817 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:07:26.443829 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:07:29.450033 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:07:29.450135 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:07:29.450170 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:07:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:07:36.411974 3798194 memory.go:184] no items to output this cycle I1121 19:07:36.412138 3798194 cpu.go:275] no items to output this cycle E1121 19:07:46.413399 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:07:46.413432 3798194 memory.go:184] no items to output this cycle I1121 19:07:46.413579 3798194 cpu.go:275] no items to output this cycle E1121 19:07:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:07:56.413269 3798194 memory.go:191] Add success. W1121 19:07:56.413301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:07:56.413313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:07:56.413316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:07:56.417631 3798194 cpu.go:282] Add success. I1121 19:07:56.440968 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:07:56.441553 3798194 net.go:642] Add success. I1121 19:07:56.443071 3798194 net.go:778] primary dev: ETH0 I1121 19:07:56.443087 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:07:56.443100 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:07:57.451344 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:07:57.451516 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:07:57.451573 3798194 disk_worker.go:723] disk space is not compliant W1121 19:07:57.451576 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:07:57.463071 3798194 disk_worker.go:509] system disk:vda1 I1121 19:07:57.463133 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:07:58.446688 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:07:59.062140 3798194 disk_info.go:125] begin check local disk info of client I1121 19:07:59.065940 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:07:59.065948 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004820c0 0xc000482100] I1121 19:07:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:07:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:07:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:07:59.471667 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:08:06.412374 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:08:06.412462 3798194 memory.go:184] no items to output this cycle I1121 19:08:06.412867 3798194 cpu.go:275] no items to output this cycle E1121 19:08:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:08:16.411982 3798194 memory.go:184] no items to output this cycle I1121 19:08:16.412149 3798194 cpu.go:275] no items to output this cycle E1121 19:08:26.414615 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:08:26.414652 3798194 memory.go:191] Add success. I1121 19:08:26.419088 3798194 cpu.go:282] Add success. I1121 19:08:26.441722 3798194 net.go:642] Add success. I1121 19:08:26.443211 3798194 net.go:778] primary dev: ETH0 I1121 19:08:26.443228 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:08:26.443240 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:08:29.450360 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:08:29.450454 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:08:29.450487 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:08:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:08:36.411975 3798194 memory.go:184] no items to output this cycle I1121 19:08:36.412121 3798194 cpu.go:275] no items to output this cycle E1121 19:08:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:08:46.412255 3798194 memory.go:184] no items to output this cycle I1121 19:08:46.412406 3798194 cpu.go:275] no items to output this cycle E1121 19:08:56.413012 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:08:56.413045 3798194 memory.go:191] Add success. W1121 19:08:56.413073 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:08:56.413085 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:08:56.413089 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:08:56.417202 3798194 cpu.go:282] Add success. I1121 19:08:56.441577 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:08:56.442039 3798194 net.go:642] Add success. I1121 19:08:56.443635 3798194 net.go:778] primary dev: ETH0 I1121 19:08:56.443650 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:08:56.443663 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:08:57.450620 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:08:57.450832 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:08:57.450842 3798194 disk_worker.go:723] disk space is not compliant W1121 19:08:57.450845 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:08:57.459220 3798194 disk_worker.go:509] system disk:vda1 I1121 19:08:57.459260 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:08:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:08:59.066052 3798194 disk_info.go:125] begin check local disk info of client I1121 19:08:59.069908 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:08:59.069917 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037f240 0xc00037f280] I1121 19:08:59.450406 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:08:59.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:08:59.450509 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:08:59.471034 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:09:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:09:06.412256 3798194 memory.go:184] no items to output this cycle I1121 19:09:06.412400 3798194 cpu.go:275] no items to output this cycle E1121 19:09:16.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:09:16.413257 3798194 memory.go:184] no items to output this cycle I1121 19:09:16.413400 3798194 cpu.go:275] no items to output this cycle I1121 19:09:17.542363 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:09:17.542372 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:09:26.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:09:26.413271 3798194 memory.go:191] Add success. I1121 19:09:26.417707 3798194 cpu.go:282] Add success. I1121 19:09:26.442442 3798194 net.go:642] Add success. I1121 19:09:26.443911 3798194 net.go:778] primary dev: ETH0 I1121 19:09:26.443927 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:09:26.443946 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:09:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:09:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:09:29.450147 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:09:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:09:36.411977 3798194 memory.go:184] no items to output this cycle I1121 19:09:36.412130 3798194 cpu.go:275] no items to output this cycle E1121 19:09:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:09:46.413253 3798194 memory.go:184] no items to output this cycle I1121 19:09:46.413397 3798194 cpu.go:275] no items to output this cycle E1121 19:09:56.412289 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:09:56.412334 3798194 memory.go:191] Add success. W1121 19:09:56.412370 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:09:56.412387 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:09:56.412393 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:09:56.416877 3798194 cpu.go:282] Add success. I1121 19:09:56.442018 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:09:56.442943 3798194 net.go:642] Add success. I1121 19:09:56.444610 3798194 net.go:778] primary dev: ETH0 I1121 19:09:56.444633 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:09:56.444652 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:09:57.450626 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:09:57.450785 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:09:57.450835 3798194 disk_worker.go:723] disk space is not compliant W1121 19:09:57.450838 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:09:57.459165 3798194 disk_worker.go:509] system disk:vda1 I1121 19:09:57.459205 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:09:58.446011 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:09:59.072141 3798194 disk_info.go:125] begin check local disk info of client I1121 19:09:59.075978 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:09:59.075986 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003bd680 0xc0003bd6c0] I1121 19:09:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:09:59.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:09:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:09:59.471645 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:10:06.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:10:06.413258 3798194 memory.go:184] no items to output this cycle I1121 19:10:06.413403 3798194 cpu.go:275] no items to output this cycle E1121 19:10:16.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:10:16.411988 3798194 memory.go:184] no items to output this cycle I1121 19:10:16.412138 3798194 cpu.go:275] no items to output this cycle E1121 19:10:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:10:26.411953 3798194 memory.go:191] Add success. I1121 19:10:26.412315 3798194 cpu.go:282] Add success. I1121 19:10:26.441761 3798194 net.go:642] Add success. I1121 19:10:26.445671 3798194 net.go:778] primary dev: ETH0 I1121 19:10:26.445686 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:10:26.445697 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:10:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:10:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:10:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:10:36.412929 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:10:36.412952 3798194 memory.go:184] no items to output this cycle I1121 19:10:36.413098 3798194 cpu.go:275] no items to output this cycle E1121 19:10:46.413042 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:10:46.413064 3798194 memory.go:184] no items to output this cycle I1121 19:10:46.413240 3798194 cpu.go:275] no items to output this cycle E1121 19:10:56.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:10:56.412278 3798194 memory.go:191] Add success. W1121 19:10:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:10:56.412315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:10:56.412318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:10:56.416521 3798194 cpu.go:282] Add success. I1121 19:10:56.444745 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:10:56.445243 3798194 net.go:642] Add success. I1121 19:10:56.446695 3798194 net.go:778] primary dev: ETH0 I1121 19:10:56.446709 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:10:56.446722 3798194 net.go:659] Add eni metrics success, count: 7 E1121 19:10:57.450951 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 19:10:57.450965 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:10:57.451172 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:10:57.451182 3798194 disk_worker.go:723] disk space is not compliant W1121 19:10:57.451185 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:10:57.459924 3798194 disk_worker.go:509] system disk:vda1 I1121 19:10:57.459965 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 19:10:58.448427 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 19:10:58.448442 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:10:59.078145 3798194 disk_info.go:125] begin check local disk info of client I1121 19:10:59.081934 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:10:59.081942 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0f80 0xc0002e0fc0] E1121 19:10:59.448973 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 19:10:59.450046 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:10:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:10:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:10:59.471648 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:11:06.413130 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:11:06.413154 3798194 memory.go:184] no items to output this cycle I1121 19:11:06.413303 3798194 cpu.go:275] no items to output this cycle E1121 19:11:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:11:16.411974 3798194 memory.go:184] no items to output this cycle I1121 19:11:16.412131 3798194 cpu.go:275] no items to output this cycle E1121 19:11:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:11:26.413263 3798194 memory.go:191] Add success. I1121 19:11:26.417420 3798194 cpu.go:282] Add success. I1121 19:11:26.441934 3798194 net.go:642] Add success. I1121 19:11:26.443287 3798194 net.go:778] primary dev: ETH0 I1121 19:11:26.443301 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:11:26.443313 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:11:29.450130 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:11:29.450206 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:11:29.450235 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:11:36.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:11:36.411975 3798194 memory.go:184] no items to output this cycle I1121 19:11:36.412136 3798194 cpu.go:275] no items to output this cycle E1121 19:11:46.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:11:46.412248 3798194 memory.go:184] no items to output this cycle I1121 19:11:46.412392 3798194 cpu.go:275] no items to output this cycle E1121 19:11:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:11:56.412267 3798194 memory.go:191] Add success. W1121 19:11:56.412294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:11:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:11:56.412308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:11:56.416632 3798194 cpu.go:282] Add success. I1121 19:11:56.440963 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:11:56.441543 3798194 net.go:642] Add success. I1121 19:11:56.443144 3798194 net.go:778] primary dev: ETH0 I1121 19:11:56.443159 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:11:56.443177 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:11:57.450265 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:11:57.450599 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:11:57.450610 3798194 disk_worker.go:723] disk space is not compliant W1121 19:11:57.450614 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:11:57.460024 3798194 disk_worker.go:509] system disk:vda1 I1121 19:11:57.460065 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:11:58.445696 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:11:59.082048 3798194 disk_info.go:125] begin check local disk info of client I1121 19:11:59.085931 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:11:59.085940 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004778c0 0xc000477900] I1121 19:11:59.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:11:59.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:11:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:11:59.471657 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:12:06.412939 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:12:06.412961 3798194 memory.go:184] no items to output this cycle I1121 19:12:06.413102 3798194 cpu.go:275] no items to output this cycle E1121 19:12:16.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:12:16.412264 3798194 memory.go:184] no items to output this cycle I1121 19:12:16.412408 3798194 cpu.go:275] no items to output this cycle I1121 19:12:17.542533 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:12:17.542542 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:12:26.411197 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:12:26.411230 3798194 memory.go:191] Add success. I1121 19:12:26.411573 3798194 cpu.go:282] Add success. I1121 19:12:26.442247 3798194 net.go:642] Add success. I1121 19:12:26.446188 3798194 net.go:778] primary dev: ETH0 I1121 19:12:26.446203 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:12:26.446216 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:12:29.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:12:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:12:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:12:36.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:12:36.412250 3798194 memory.go:184] no items to output this cycle I1121 19:12:36.412393 3798194 cpu.go:275] no items to output this cycle E1121 19:12:46.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:12:46.413253 3798194 memory.go:184] no items to output this cycle I1121 19:12:46.413401 3798194 cpu.go:275] no items to output this cycle E1121 19:12:56.411371 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:12:56.411405 3798194 memory.go:191] Add success. W1121 19:12:56.411430 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:12:56.411441 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:12:56.411445 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:12:56.415522 3798194 cpu.go:282] Add success. I1121 19:12:56.441890 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:12:56.442417 3798194 net.go:642] Add success. I1121 19:12:56.444010 3798194 net.go:778] primary dev: ETH0 I1121 19:12:56.444026 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:12:56.444039 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:12:57.450756 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:12:57.450950 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:12:57.450961 3798194 disk_worker.go:723] disk space is not compliant W1121 19:12:57.450964 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:12:57.459332 3798194 disk_worker.go:509] system disk:vda1 I1121 19:12:57.459374 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:12:58.445654 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:12:59.086037 3798194 disk_info.go:125] begin check local disk info of client I1121 19:12:59.089867 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:12:59.089876 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003acb80 0xc0003acbc0] I1121 19:12:59.450413 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:12:59.450493 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:12:59.450518 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:12:59.471022 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:13:06.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:13:06.413250 3798194 memory.go:184] no items to output this cycle I1121 19:13:06.413392 3798194 cpu.go:275] no items to output this cycle E1121 19:13:16.413254 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:13:16.413377 3798194 memory.go:184] no items to output this cycle I1121 19:13:16.413703 3798194 cpu.go:275] no items to output this cycle E1121 19:13:26.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:13:26.412265 3798194 memory.go:191] Add success. I1121 19:13:26.416858 3798194 cpu.go:282] Add success. I1121 19:13:26.441518 3798194 net.go:642] Add success. I1121 19:13:26.443081 3798194 net.go:778] primary dev: ETH0 I1121 19:13:26.443095 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:13:26.443120 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:13:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:13:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:13:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:13:36.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:13:36.413254 3798194 memory.go:184] no items to output this cycle I1121 19:13:36.413396 3798194 cpu.go:275] no items to output this cycle E1121 19:13:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:13:46.413256 3798194 memory.go:184] no items to output this cycle I1121 19:13:46.413412 3798194 cpu.go:275] no items to output this cycle E1121 19:13:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:13:56.411949 3798194 memory.go:191] Add success. W1121 19:13:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:13:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:13:56.411991 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:13:56.412346 3798194 cpu.go:282] Add success. I1121 19:13:56.441810 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:13:56.442321 3798194 net.go:642] Add success. I1121 19:13:56.446380 3798194 net.go:778] primary dev: ETH0 I1121 19:13:56.446397 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:13:56.446408 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:13:57.450608 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:13:57.450779 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:13:57.450831 3798194 disk_worker.go:723] disk space is not compliant W1121 19:13:57.450835 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:13:57.459158 3798194 disk_worker.go:509] system disk:vda1 I1121 19:13:57.459198 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:13:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:13:59.089985 3798194 disk_info.go:125] begin check local disk info of client I1121 19:13:59.093814 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:13:59.093823 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002fff00 0xc0002fff40] I1121 19:13:59.450415 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:13:59.450495 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:13:59.450522 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:13:59.471037 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:14:06.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:14:06.413268 3798194 memory.go:184] no items to output this cycle I1121 19:14:06.413409 3798194 cpu.go:275] no items to output this cycle E1121 19:14:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:14:16.411991 3798194 memory.go:184] no items to output this cycle I1121 19:14:16.412153 3798194 cpu.go:275] no items to output this cycle E1121 19:14:26.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:14:26.412274 3798194 memory.go:191] Add success. I1121 19:14:26.416656 3798194 cpu.go:282] Add success. I1121 19:14:26.442204 3798194 net.go:642] Add success. I1121 19:14:26.443579 3798194 net.go:778] primary dev: ETH0 I1121 19:14:26.443594 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:14:26.443608 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:14:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:14:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:14:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:14:36.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:14:36.413247 3798194 memory.go:184] no items to output this cycle I1121 19:14:36.413546 3798194 cpu.go:275] no items to output this cycle E1121 19:14:46.412590 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:14:46.412612 3798194 memory.go:184] no items to output this cycle I1121 19:14:46.412760 3798194 cpu.go:275] no items to output this cycle E1121 19:14:56.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:14:56.413283 3798194 memory.go:191] Add success. W1121 19:14:56.413318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:14:56.413330 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:14:56.413333 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:14:56.417665 3798194 cpu.go:282] Add success. I1121 19:14:56.440960 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:14:56.441562 3798194 net.go:642] Add success. I1121 19:14:56.442978 3798194 net.go:778] primary dev: ETH0 I1121 19:14:56.442993 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:14:56.443007 3798194 net.go:659] Add eni metrics success, count: 7 W1121 19:14:57.443909 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:14:57.443984 3798194 disk_worker.go:723] disk space is not compliant W1121 19:14:57.443988 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:14:57.452524 3798194 custom_config.go:106] query custom config with name: gpu I1121 19:14:57.452552 3798194 disk_worker.go:509] system disk:vda1 I1121 19:14:57.452595 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:14:58.446014 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:14:59.096149 3798194 disk_info.go:125] begin check local disk info of client I1121 19:14:59.099882 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:14:59.099891 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f83c0 0xc0003f8400] I1121 19:14:59.450414 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:14:59.450491 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:14:59.450515 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:14:59.471019 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:15:06.412322 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:15:06.412343 3798194 memory.go:184] no items to output this cycle I1121 19:15:06.412482 3798194 cpu.go:275] no items to output this cycle E1121 19:15:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:15:16.411976 3798194 memory.go:184] no items to output this cycle I1121 19:15:16.412146 3798194 cpu.go:275] no items to output this cycle I1121 19:15:17.542709 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:15:17.542717 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:15:26.411227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:15:26.411260 3798194 memory.go:191] Add success. I1121 19:15:26.411623 3798194 cpu.go:282] Add success. I1121 19:15:26.442215 3798194 net.go:642] Add success. I1121 19:15:26.446158 3798194 net.go:778] primary dev: ETH0 I1121 19:15:26.446173 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:15:26.446188 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:15:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:15:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:15:29.450496 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:15:36.412298 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:15:36.412322 3798194 memory.go:184] no items to output this cycle I1121 19:15:36.412465 3798194 cpu.go:275] no items to output this cycle E1121 19:15:46.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:15:46.411968 3798194 memory.go:184] no items to output this cycle I1121 19:15:46.412121 3798194 cpu.go:275] no items to output this cycle E1121 19:15:56.412875 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:15:56.412912 3798194 memory.go:191] Add success. W1121 19:15:56.412941 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:15:56.412953 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:15:56.412956 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:15:56.416951 3798194 cpu.go:282] Add success. I1121 19:15:56.441503 3798194 event_worker.go:152] Polling the log file for events... I1121 19:15:56.444659 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:15:56.445182 3798194 net.go:642] Add success. I1121 19:15:56.446696 3798194 net.go:778] primary dev: ETH0 I1121 19:15:56.446711 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:15:56.446722 3798194 net.go:659] Add eni metrics success, count: 7 E1121 19:15:57.451233 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 19:15:57.451250 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:15:57.451566 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:15:57.451632 3798194 disk_worker.go:723] disk space is not compliant W1121 19:15:57.451636 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:15:57.461029 3798194 disk_worker.go:509] system disk:vda1 I1121 19:15:57.461070 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 19:15:58.447877 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 19:15:58.447891 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:15:59.103147 3798194 disk_info.go:125] begin check local disk info of client I1121 19:15:59.107002 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:15:59.107010 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00044b980 0xc00044b9c0] E1121 19:15:59.448964 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 19:15:59.450044 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:15:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:15:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:15:59.471653 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:16:06.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:16:06.413264 3798194 memory.go:184] no items to output this cycle I1121 19:16:06.413416 3798194 cpu.go:275] no items to output this cycle E1121 19:16:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:16:16.412263 3798194 memory.go:184] no items to output this cycle I1121 19:16:16.412404 3798194 cpu.go:275] no items to output this cycle E1121 19:16:26.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:16:26.413278 3798194 memory.go:191] Add success. I1121 19:16:26.417501 3798194 cpu.go:282] Add success. I1121 19:16:26.442300 3798194 net.go:642] Add success. I1121 19:16:26.443712 3798194 net.go:778] primary dev: ETH0 I1121 19:16:26.443727 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:16:26.443742 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:16:29.450511 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:16:29.450591 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:16:29.450619 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:16:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:16:36.412261 3798194 memory.go:184] no items to output this cycle I1121 19:16:36.412409 3798194 cpu.go:275] no items to output this cycle E1121 19:16:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:16:46.413260 3798194 memory.go:184] no items to output this cycle I1121 19:16:46.413408 3798194 cpu.go:275] no items to output this cycle E1121 19:16:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:16:56.412267 3798194 memory.go:191] Add success. W1121 19:16:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:16:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:16:56.412317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:16:56.416671 3798194 cpu.go:282] Add success. I1121 19:16:56.440951 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:16:56.441530 3798194 net.go:642] Add success. I1121 19:16:56.443043 3798194 net.go:778] primary dev: ETH0 I1121 19:16:56.443058 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:16:56.443070 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:16:57.451137 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:16:57.451352 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:16:57.451363 3798194 disk_worker.go:723] disk space is not compliant W1121 19:16:57.451366 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:16:57.459815 3798194 disk_worker.go:509] system disk:vda1 I1121 19:16:57.459858 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:16:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:16:59.107107 3798194 disk_info.go:125] begin check local disk info of client I1121 19:16:59.110783 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:16:59.110791 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f6c40 0xc0003f6c80] I1121 19:16:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:16:59.450083 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:16:59.450116 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:16:59.471624 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:17:06.412588 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:17:06.412613 3798194 memory.go:184] no items to output this cycle I1121 19:17:06.412755 3798194 cpu.go:275] no items to output this cycle E1121 19:17:16.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:17:16.411987 3798194 memory.go:184] no items to output this cycle I1121 19:17:16.412145 3798194 cpu.go:275] no items to output this cycle E1121 19:17:26.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:17:26.411965 3798194 memory.go:191] Add success. I1121 19:17:26.412354 3798194 cpu.go:282] Add success. I1121 19:17:26.441940 3798194 net.go:642] Add success. I1121 19:17:26.445967 3798194 net.go:778] primary dev: ETH0 I1121 19:17:26.445982 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:17:26.445995 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:17:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:17:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:17:29.450157 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:17:36.412762 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:17:36.412782 3798194 memory.go:184] no items to output this cycle I1121 19:17:36.412936 3798194 cpu.go:275] no items to output this cycle E1121 19:17:46.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:17:46.413253 3798194 memory.go:184] no items to output this cycle I1121 19:17:46.413401 3798194 cpu.go:275] no items to output this cycle E1121 19:17:56.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:17:56.412273 3798194 memory.go:191] Add success. W1121 19:17:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:17:56.412314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:17:56.412317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:17:56.416954 3798194 cpu.go:282] Add success. I1121 19:17:56.441339 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:17:56.441900 3798194 net.go:642] Add success. I1121 19:17:56.443506 3798194 net.go:778] primary dev: ETH0 I1121 19:17:56.443529 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:17:56.443541 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:17:57.450920 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:17:57.451137 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:17:57.451149 3798194 disk_worker.go:723] disk space is not compliant W1121 19:17:57.451152 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:17:57.459886 3798194 disk_worker.go:509] system disk:vda1 I1121 19:17:57.459930 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:17:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:17:59.113757 3798194 disk_info.go:125] begin check local disk info of client I1121 19:17:59.117635 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:17:59.117644 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000310b40 0xc000310b80] I1121 19:17:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:17:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:17:59.450128 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:17:59.471662 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:18:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:18:06.411985 3798194 memory.go:184] no items to output this cycle I1121 19:18:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 19:18:16.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:18:16.412258 3798194 memory.go:184] no items to output this cycle I1121 19:18:16.412416 3798194 cpu.go:275] no items to output this cycle I1121 19:18:17.542872 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:18:17.542885 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:18:26.411483 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:18:26.411517 3798194 memory.go:191] Add success. I1121 19:18:26.411860 3798194 cpu.go:282] Add success. I1121 19:18:26.441480 3798194 net.go:642] Add success. I1121 19:18:26.445346 3798194 net.go:778] primary dev: ETH0 I1121 19:18:26.445361 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:18:26.445383 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:18:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:18:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:18:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:18:36.412494 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:18:36.412518 3798194 memory.go:184] no items to output this cycle I1121 19:18:36.412664 3798194 cpu.go:275] no items to output this cycle E1121 19:18:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:18:46.411989 3798194 memory.go:184] no items to output this cycle I1121 19:18:46.412153 3798194 cpu.go:275] no items to output this cycle E1121 19:18:56.412099 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:18:56.412158 3798194 memory.go:191] Add success. W1121 19:18:56.412188 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:18:56.412200 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:18:56.412203 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:18:56.418704 3798194 cpu.go:282] Add success. I1121 19:18:56.441727 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:18:56.442217 3798194 net.go:642] Add success. I1121 19:18:56.448139 3798194 net.go:778] primary dev: ETH0 I1121 19:18:56.448154 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:18:56.448168 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:18:57.445611 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:18:57.445870 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:18:57.445883 3798194 disk_worker.go:723] disk space is not compliant W1121 19:18:57.445887 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:18:57.450831 3798194 disk_worker.go:509] system disk:vda1 I1121 19:18:57.450870 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:18:58.445980 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:18:59.117748 3798194 disk_info.go:125] begin check local disk info of client I1121 19:18:59.121667 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:18:59.121676 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e380 0xc00030e3c0] I1121 19:18:59.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:18:59.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:18:59.450148 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:18:59.471646 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:19:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:19:06.411976 3798194 memory.go:184] no items to output this cycle I1121 19:19:06.412131 3798194 cpu.go:275] no items to output this cycle E1121 19:19:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:19:16.411983 3798194 memory.go:184] no items to output this cycle I1121 19:19:16.412143 3798194 cpu.go:275] no items to output this cycle E1121 19:19:26.412497 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:19:26.412536 3798194 memory.go:191] Add success. I1121 19:19:26.416964 3798194 cpu.go:282] Add success. I1121 19:19:26.441479 3798194 net.go:642] Add success. I1121 19:19:26.443187 3798194 net.go:778] primary dev: ETH0 I1121 19:19:26.443201 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:19:26.443214 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:19:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:19:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:19:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:19:36.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:19:36.411975 3798194 memory.go:184] no items to output this cycle I1121 19:19:36.412130 3798194 cpu.go:275] no items to output this cycle E1121 19:19:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:19:46.413263 3798194 memory.go:184] no items to output this cycle I1121 19:19:46.413410 3798194 cpu.go:275] no items to output this cycle E1121 19:19:56.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:19:56.412261 3798194 memory.go:191] Add success. W1121 19:19:56.412288 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:19:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:19:56.412303 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:19:56.416541 3798194 cpu.go:282] Add success. I1121 19:19:56.441928 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:19:56.442427 3798194 net.go:642] Add success. I1121 19:19:56.443993 3798194 net.go:778] primary dev: ETH0 I1121 19:19:56.444008 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:19:56.444021 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:19:57.450796 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:19:57.450961 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:19:57.451011 3798194 disk_worker.go:723] disk space is not compliant W1121 19:19:57.451014 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:19:57.459279 3798194 disk_worker.go:509] system disk:vda1 I1121 19:19:57.459321 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:19:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:19:59.124138 3798194 disk_info.go:125] begin check local disk info of client I1121 19:19:59.127937 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:19:59.128057 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a6000 0xc0004a6040] I1121 19:19:59.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:19:59.450490 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:19:59.450522 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:19:59.471098 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:20:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:20:06.412252 3798194 memory.go:184] no items to output this cycle I1121 19:20:06.412395 3798194 cpu.go:275] no items to output this cycle E1121 19:20:16.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:20:16.411970 3798194 memory.go:184] no items to output this cycle I1121 19:20:16.412126 3798194 cpu.go:275] no items to output this cycle E1121 19:20:26.411930 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:20:26.411965 3798194 memory.go:191] Add success. I1121 19:20:26.412310 3798194 cpu.go:282] Add success. I1121 19:20:26.441925 3798194 net.go:642] Add success. I1121 19:20:26.445900 3798194 net.go:778] primary dev: ETH0 I1121 19:20:26.445916 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:20:26.445929 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:20:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:20:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:20:29.450150 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:20:36.412755 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:20:36.412779 3798194 memory.go:184] no items to output this cycle I1121 19:20:36.412926 3798194 cpu.go:275] no items to output this cycle E1121 19:20:46.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:20:46.411984 3798194 memory.go:184] no items to output this cycle I1121 19:20:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 19:20:56.411909 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:20:56.411943 3798194 memory.go:191] Add success. W1121 19:20:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:20:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:20:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:20:56.412355 3798194 cpu.go:282] Add success. I1121 19:20:56.445095 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:20:56.445601 3798194 net.go:642] Add success. I1121 19:20:56.449796 3798194 net.go:778] primary dev: ETH0 I1121 19:20:56.449812 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:20:56.449824 3798194 net.go:659] Add eni metrics success, count: 7 E1121 19:20:57.450598 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 19:20:57.450612 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:20:57.450772 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:20:57.450828 3798194 disk_worker.go:723] disk space is not compliant W1121 19:20:57.450831 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:20:57.463935 3798194 disk_worker.go:509] system disk:vda1 I1121 19:20:57.463982 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 19:20:58.448185 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 19:20:58.448200 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:20:59.130148 3798194 disk_info.go:125] begin check local disk info of client I1121 19:20:59.133995 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:20:59.134005 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037d540 0xc00037d580] E1121 19:20:59.448622 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 19:20:59.449660 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:20:59.449713 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:20:59.449737 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:20:59.471265 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:21:06.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:21:06.412262 3798194 memory.go:184] no items to output this cycle I1121 19:21:06.412414 3798194 cpu.go:275] no items to output this cycle E1121 19:21:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:21:16.411975 3798194 memory.go:184] no items to output this cycle I1121 19:21:16.412126 3798194 cpu.go:275] no items to output this cycle I1121 19:21:17.543264 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:21:17.543273 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:21:26.411319 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:21:26.411352 3798194 memory.go:191] Add success. I1121 19:21:26.411668 3798194 cpu.go:282] Add success. I1121 19:21:26.441224 3798194 net.go:642] Add success. I1121 19:21:26.445154 3798194 net.go:778] primary dev: ETH0 I1121 19:21:26.445174 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:21:26.445213 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:21:29.450033 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:21:29.450134 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:21:29.450164 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:21:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:21:36.412260 3798194 memory.go:184] no items to output this cycle I1121 19:21:36.412400 3798194 cpu.go:275] no items to output this cycle E1121 19:21:46.412431 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:21:46.412455 3798194 memory.go:184] no items to output this cycle I1121 19:21:46.412598 3798194 cpu.go:275] no items to output this cycle E1121 19:21:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:21:56.411951 3798194 memory.go:191] Add success. W1121 19:21:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:21:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:21:56.411996 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:21:56.412349 3798194 cpu.go:282] Add success. I1121 19:21:56.441804 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:21:56.442386 3798194 net.go:642] Add success. I1121 19:21:56.446892 3798194 net.go:778] primary dev: ETH0 I1121 19:21:56.446906 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:21:56.446919 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:21:57.443680 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:21:57.445519 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:21:57.445619 3798194 disk_worker.go:723] disk space is not compliant W1121 19:21:57.445624 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:21:57.448539 3798194 disk_worker.go:509] system disk:vda1 I1121 19:21:57.448608 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:21:58.445982 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:21:59.137142 3798194 disk_info.go:125] begin check local disk info of client I1121 19:21:59.140762 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:21:59.140773 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000368e00 0xc000368e40] I1121 19:21:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:21:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:21:59.450144 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:21:59.471644 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:22:06.413166 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:22:06.413192 3798194 memory.go:184] no items to output this cycle I1121 19:22:06.413339 3798194 cpu.go:275] no items to output this cycle E1121 19:22:16.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:22:16.412259 3798194 memory.go:184] no items to output this cycle I1121 19:22:16.412404 3798194 cpu.go:275] no items to output this cycle E1121 19:22:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:22:26.411951 3798194 memory.go:191] Add success. I1121 19:22:26.412306 3798194 cpu.go:282] Add success. I1121 19:22:26.441929 3798194 net.go:642] Add success. I1121 19:22:26.446416 3798194 net.go:778] primary dev: ETH0 I1121 19:22:26.446432 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:22:26.446445 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:22:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:22:29.450487 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:22:29.450515 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:22:36.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:22:36.412253 3798194 memory.go:184] no items to output this cycle I1121 19:22:36.412395 3798194 cpu.go:275] no items to output this cycle E1121 19:22:46.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:22:46.412875 3798194 memory.go:184] no items to output this cycle I1121 19:22:46.413019 3798194 cpu.go:275] no items to output this cycle E1121 19:22:56.412180 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:22:56.412212 3798194 memory.go:191] Add success. W1121 19:22:56.412239 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:22:56.412251 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:22:56.412254 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:22:56.416393 3798194 cpu.go:282] Add success. I1121 19:22:56.441788 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:22:56.442689 3798194 net.go:642] Add success. I1121 19:22:56.444212 3798194 net.go:778] primary dev: ETH0 I1121 19:22:56.444228 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:22:56.444243 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:22:57.450885 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:22:57.451098 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:22:57.451124 3798194 disk_worker.go:723] disk space is not compliant W1121 19:22:57.451129 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:22:57.459905 3798194 disk_worker.go:509] system disk:vda1 I1121 19:22:57.459946 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:22:58.446133 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:22:59.140878 3798194 disk_info.go:125] begin check local disk info of client I1121 19:22:59.144721 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:22:59.144730 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00049d740 0xc00049d780] I1121 19:22:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:22:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:22:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:22:59.471634 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:23:06.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:23:06.412258 3798194 memory.go:184] no items to output this cycle I1121 19:23:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 19:23:16.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:23:16.413253 3798194 memory.go:184] no items to output this cycle I1121 19:23:16.413398 3798194 cpu.go:275] no items to output this cycle E1121 19:23:26.412784 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:23:26.412820 3798194 memory.go:191] Add success. I1121 19:23:26.417371 3798194 cpu.go:282] Add success. I1121 19:23:26.441889 3798194 net.go:642] Add success. I1121 19:23:26.443440 3798194 net.go:778] primary dev: ETH0 I1121 19:23:26.443461 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:23:26.443475 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:23:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:23:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:23:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:23:36.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:23:36.411974 3798194 memory.go:184] no items to output this cycle I1121 19:23:36.412120 3798194 cpu.go:275] no items to output this cycle E1121 19:23:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:23:46.411980 3798194 memory.go:184] no items to output this cycle I1121 19:23:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 19:23:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:23:56.411948 3798194 memory.go:191] Add success. W1121 19:23:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:23:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:23:56.411996 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:23:56.412333 3798194 cpu.go:282] Add success. I1121 19:23:56.441804 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:23:56.442311 3798194 net.go:642] Add success. I1121 19:23:56.446060 3798194 net.go:778] primary dev: ETH0 I1121 19:23:56.446074 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:23:56.446086 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:23:57.451064 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:23:57.451300 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:23:57.451313 3798194 disk_worker.go:723] disk space is not compliant W1121 19:23:57.451316 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:23:57.460101 3798194 disk_worker.go:509] system disk:vda1 I1121 19:23:57.460161 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:23:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:23:59.147527 3798194 disk_info.go:125] begin check local disk info of client I1121 19:23:59.151060 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:23:59.151070 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000402780 0xc0004027c0] I1121 19:23:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:23:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:23:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:23:59.471688 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:24:06.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:24:06.413249 3798194 memory.go:184] no items to output this cycle I1121 19:24:06.413395 3798194 cpu.go:275] no items to output this cycle E1121 19:24:16.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:24:16.413280 3798194 memory.go:184] no items to output this cycle I1121 19:24:16.413426 3798194 cpu.go:275] no items to output this cycle I1121 19:24:17.543433 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:24:17.543442 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:24:26.411306 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:24:26.411342 3798194 memory.go:191] Add success. I1121 19:24:26.411717 3798194 cpu.go:282] Add success. I1121 19:24:26.441305 3798194 net.go:642] Add success. I1121 19:24:26.445102 3798194 net.go:778] primary dev: ETH0 I1121 19:24:26.445135 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:24:26.445148 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:24:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:24:29.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:24:29.450153 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:24:36.412361 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:24:36.412386 3798194 memory.go:184] no items to output this cycle I1121 19:24:36.412530 3798194 cpu.go:275] no items to output this cycle E1121 19:24:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:24:46.411974 3798194 memory.go:184] no items to output this cycle I1121 19:24:46.412135 3798194 cpu.go:275] no items to output this cycle E1121 19:24:56.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:24:56.413275 3798194 memory.go:191] Add success. W1121 19:24:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:24:56.413315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:24:56.413318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:24:56.424929 3798194 cpu.go:282] Add success. I1121 19:24:56.441430 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:24:56.441923 3798194 net.go:642] Add success. I1121 19:24:56.443417 3798194 net.go:778] primary dev: ETH0 I1121 19:24:56.443431 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:24:56.443444 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:24:57.450383 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:24:57.450585 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:24:57.450596 3798194 disk_worker.go:723] disk space is not compliant W1121 19:24:57.450599 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:24:57.458987 3798194 disk_worker.go:509] system disk:vda1 I1121 19:24:57.459032 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:24:58.446565 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:24:59.153389 3798194 disk_info.go:125] begin check local disk info of client I1121 19:24:59.157320 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:24:59.157330 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037e200 0xc00037e240] I1121 19:24:59.450415 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:24:59.450491 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:24:59.450518 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:24:59.471040 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:25:06.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:25:06.413249 3798194 memory.go:184] no items to output this cycle I1121 19:25:06.413402 3798194 cpu.go:275] no items to output this cycle E1121 19:25:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:25:16.411977 3798194 memory.go:184] no items to output this cycle I1121 19:25:16.412137 3798194 cpu.go:275] no items to output this cycle E1121 19:25:26.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:25:26.413266 3798194 memory.go:191] Add success. I1121 19:25:26.417505 3798194 cpu.go:282] Add success. I1121 19:25:26.442022 3798194 net.go:642] Add success. I1121 19:25:26.443586 3798194 net.go:778] primary dev: ETH0 I1121 19:25:26.443600 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:25:26.443612 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:25:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:25:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:25:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:25:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:25:36.413260 3798194 memory.go:184] no items to output this cycle I1121 19:25:36.413405 3798194 cpu.go:275] no items to output this cycle E1121 19:25:46.412081 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:25:46.412184 3798194 memory.go:184] no items to output this cycle I1121 19:25:46.412525 3798194 cpu.go:275] no items to output this cycle E1121 19:25:56.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:25:56.412261 3798194 memory.go:191] Add success. W1121 19:25:56.412287 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:25:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:25:56.412302 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:25:56.416786 3798194 cpu.go:282] Add success. I1121 19:25:56.443852 3798194 event_worker.go:152] Polling the log file for events... I1121 19:25:56.444064 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:25:56.444590 3798194 net.go:642] Add success. I1121 19:25:56.446055 3798194 net.go:778] primary dev: ETH0 I1121 19:25:56.446070 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:25:56.446084 3798194 net.go:659] Add eni metrics success, count: 7 E1121 19:25:57.450936 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 19:25:57.450950 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:25:57.451102 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:25:57.451178 3798194 disk_worker.go:723] disk space is not compliant W1121 19:25:57.451182 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:25:57.459925 3798194 disk_worker.go:509] system disk:vda1 I1121 19:25:57.459968 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 19:25:58.448485 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 19:25:58.448501 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:25:59.159155 3798194 disk_info.go:125] begin check local disk info of client I1121 19:25:59.162915 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:25:59.162924 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000464480 0xc0004644c0] E1121 19:25:59.448979 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 19:25:59.450056 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:25:59.450116 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:25:59.450147 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:25:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:26:06.413670 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:26:06.413692 3798194 memory.go:184] no items to output this cycle I1121 19:26:06.413845 3798194 cpu.go:275] no items to output this cycle E1121 19:26:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:26:16.411974 3798194 memory.go:184] no items to output this cycle I1121 19:26:16.412139 3798194 cpu.go:275] no items to output this cycle E1121 19:26:26.411930 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:26:26.411961 3798194 memory.go:191] Add success. I1121 19:26:26.412313 3798194 cpu.go:282] Add success. I1121 19:26:26.442019 3798194 net.go:642] Add success. I1121 19:26:26.445980 3798194 net.go:778] primary dev: ETH0 I1121 19:26:26.445993 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:26:26.446005 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:26:29.450006 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:26:29.450084 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:26:29.450119 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:26:36.413219 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:26:36.413242 3798194 memory.go:184] no items to output this cycle I1121 19:26:36.413388 3798194 cpu.go:275] no items to output this cycle E1121 19:26:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:26:46.412258 3798194 memory.go:184] no items to output this cycle I1121 19:26:46.412416 3798194 cpu.go:275] no items to output this cycle E1121 19:26:56.412222 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:26:56.412255 3798194 memory.go:191] Add success. W1121 19:26:56.412291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:26:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:26:56.412306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:26:56.416471 3798194 cpu.go:282] Add success. I1121 19:26:56.441878 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:26:56.442393 3798194 net.go:642] Add success. I1121 19:26:56.443976 3798194 net.go:778] primary dev: ETH0 I1121 19:26:56.443990 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:26:56.444005 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:26:57.460761 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:26:57.460924 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:26:57.460981 3798194 disk_worker.go:723] disk space is not compliant W1121 19:26:57.460984 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:26:57.474035 3798194 disk_worker.go:509] system disk:vda1 I1121 19:26:57.474077 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:26:58.446368 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:26:59.163027 3798194 disk_info.go:125] begin check local disk info of client I1121 19:26:59.166910 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:26:59.166920 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000481580 0xc0004815c0] I1121 19:26:59.450411 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:26:59.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:26:59.450511 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:26:59.471037 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:27:06.412463 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:27:06.412484 3798194 memory.go:184] no items to output this cycle I1121 19:27:06.412628 3798194 cpu.go:275] no items to output this cycle E1121 19:27:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:27:16.411985 3798194 memory.go:184] no items to output this cycle I1121 19:27:16.412144 3798194 cpu.go:275] no items to output this cycle I1121 19:27:17.543598 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:27:17.543607 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:27:26.411162 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:27:26.411196 3798194 memory.go:191] Add success. I1121 19:27:26.411508 3798194 cpu.go:282] Add success. I1121 19:27:26.442233 3798194 net.go:642] Add success. I1121 19:27:26.446274 3798194 net.go:778] primary dev: ETH0 I1121 19:27:26.446289 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:27:26.446303 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:27:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:27:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:27:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:27:36.412577 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:27:36.412602 3798194 memory.go:184] no items to output this cycle I1121 19:27:36.412746 3798194 cpu.go:275] no items to output this cycle E1121 19:27:46.411945 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:27:46.411968 3798194 memory.go:184] no items to output this cycle I1121 19:27:46.412129 3798194 cpu.go:275] no items to output this cycle E1121 19:27:56.411910 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:27:56.411940 3798194 memory.go:191] Add success. W1121 19:27:56.411969 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:27:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:27:56.411983 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:27:56.412340 3798194 cpu.go:282] Add success. I1121 19:27:56.441242 3798194 net.go:642] Add success. I1121 19:27:56.446005 3798194 net.go:778] primary dev: ETH0 I1121 19:27:56.446020 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:27:56.446032 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:27:56.441637 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:27:57.445467 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:27:57.445671 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:27:57.445731 3798194 disk_worker.go:723] disk space is not compliant W1121 19:27:57.445734 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:27:57.447825 3798194 disk_worker.go:509] system disk:vda1 I1121 19:27:57.447865 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:27:58.445986 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:27:59.167022 3798194 disk_info.go:125] begin check local disk info of client I1121 19:27:59.170892 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:27:59.170901 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b8a40 0xc0003b8a80] I1121 19:27:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:27:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:27:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:27:59.471653 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:28:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:28:06.411976 3798194 memory.go:184] no items to output this cycle I1121 19:28:06.412132 3798194 cpu.go:275] no items to output this cycle E1121 19:28:16.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:28:16.411971 3798194 memory.go:184] no items to output this cycle I1121 19:28:16.412148 3798194 cpu.go:275] no items to output this cycle E1121 19:28:26.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:28:26.412986 3798194 memory.go:191] Add success. I1121 19:28:26.417948 3798194 cpu.go:282] Add success. I1121 19:28:26.441584 3798194 net.go:642] Add success. I1121 19:28:26.442995 3798194 net.go:778] primary dev: ETH0 I1121 19:28:26.443010 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:28:26.443024 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:28:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:28:29.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:28:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:28:36.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:28:36.412871 3798194 memory.go:184] no items to output this cycle I1121 19:28:36.413013 3798194 cpu.go:275] no items to output this cycle E1121 19:28:46.413249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:28:46.413272 3798194 memory.go:184] no items to output this cycle I1121 19:28:46.413415 3798194 cpu.go:275] no items to output this cycle E1121 19:28:56.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:28:56.411989 3798194 memory.go:191] Add success. W1121 19:28:56.412016 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:28:56.412029 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:28:56.412032 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:28:56.416750 3798194 cpu.go:282] Add success. I1121 19:28:56.441155 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:28:56.441634 3798194 net.go:642] Add success. I1121 19:28:56.443085 3798194 net.go:778] primary dev: ETH0 I1121 19:28:56.443100 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:28:56.443128 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:28:57.450389 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:28:57.450599 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:28:57.450610 3798194 disk_worker.go:723] disk space is not compliant W1121 19:28:57.450613 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:28:57.459058 3798194 disk_worker.go:509] system disk:vda1 I1121 19:28:57.459104 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:28:58.445939 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:28:59.173607 3798194 disk_info.go:125] begin check local disk info of client I1121 19:28:59.177492 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:28:59.177500 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00027e780 0xc00027e7c0] I1121 19:28:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:28:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:28:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:28:59.471673 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:29:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:29:06.411975 3798194 memory.go:184] no items to output this cycle I1121 19:29:06.412138 3798194 cpu.go:275] no items to output this cycle E1121 19:29:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:29:16.412260 3798194 memory.go:184] no items to output this cycle I1121 19:29:16.412403 3798194 cpu.go:275] no items to output this cycle E1121 19:29:26.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:29:26.412266 3798194 memory.go:191] Add success. I1121 19:29:26.416566 3798194 cpu.go:282] Add success. I1121 19:29:26.442088 3798194 net.go:642] Add success. I1121 19:29:26.443635 3798194 net.go:778] primary dev: ETH0 I1121 19:29:26.443649 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:29:26.443663 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:29:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:29:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:29:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:29:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:29:36.411976 3798194 memory.go:184] no items to output this cycle I1121 19:29:36.412137 3798194 cpu.go:275] no items to output this cycle E1121 19:29:46.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:29:46.411969 3798194 memory.go:184] no items to output this cycle I1121 19:29:46.412129 3798194 cpu.go:275] no items to output this cycle E1121 19:29:56.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:29:56.411962 3798194 memory.go:191] Add success. W1121 19:29:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:29:56.412001 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:29:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:29:56.412386 3798194 cpu.go:282] Add success. I1121 19:29:56.441834 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:29:56.442430 3798194 net.go:642] Add success. I1121 19:29:56.446528 3798194 net.go:778] primary dev: ETH0 I1121 19:29:56.446544 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:29:56.446557 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:29:57.450263 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:29:57.450459 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:29:57.450470 3798194 disk_worker.go:723] disk space is not compliant W1121 19:29:57.450473 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:29:57.458509 3798194 disk_worker.go:509] system disk:vda1 I1121 19:29:57.458551 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:29:58.446485 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:29:59.181184 3798194 disk_info.go:125] begin check local disk info of client I1121 19:29:59.185033 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:29:59.185042 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e00c0 0xc0002e0100] I1121 19:29:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:29:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:29:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:29:59.471659 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:30:06.412100 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:30:06.412141 3798194 memory.go:184] no items to output this cycle I1121 19:30:06.412286 3798194 cpu.go:275] no items to output this cycle E1121 19:30:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:30:16.411984 3798194 memory.go:184] no items to output this cycle I1121 19:30:16.412148 3798194 cpu.go:275] no items to output this cycle I1121 19:30:17.544565 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:30:17.544573 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:30:26.411452 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:30:26.411488 3798194 memory.go:191] Add success. I1121 19:30:26.411828 3798194 cpu.go:282] Add success. I1121 19:30:26.441495 3798194 net.go:642] Add success. I1121 19:30:26.445590 3798194 net.go:778] primary dev: ETH0 I1121 19:30:26.445605 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:30:26.445618 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:30:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:30:29.450117 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:30:29.450152 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:30:36.412503 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:30:36.412527 3798194 memory.go:184] no items to output this cycle I1121 19:30:36.412677 3798194 cpu.go:275] no items to output this cycle E1121 19:30:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:30:46.411977 3798194 memory.go:184] no items to output this cycle I1121 19:30:46.412139 3798194 cpu.go:275] no items to output this cycle E1121 19:30:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:30:56.411947 3798194 memory.go:191] Add success. W1121 19:30:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:30:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:30:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:30:56.412315 3798194 cpu.go:282] Add success. I1121 19:30:56.453159 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:30:56.453767 3798194 net.go:642] Add success. I1121 19:30:56.457960 3798194 net.go:778] primary dev: ETH0 I1121 19:30:56.457977 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:30:56.457991 3798194 net.go:659] Add eni metrics success, count: 7 E1121 19:30:57.451334 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 19:30:57.451348 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:30:57.451562 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:30:57.451571 3798194 disk_worker.go:723] disk space is not compliant W1121 19:30:57.451575 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:30:57.460003 3798194 disk_worker.go:509] system disk:vda1 I1121 19:30:57.460059 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 19:30:58.448452 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 19:30:58.448467 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:30:59.188144 3798194 disk_info.go:125] begin check local disk info of client I1121 19:30:59.191992 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:30:59.192001 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000452380 0xc0004523c0] E1121 19:30:59.448611 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 19:30:59.449686 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:30:59.449736 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:30:59.449755 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:30:59.471297 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:31:06.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:31:06.411986 3798194 memory.go:184] no items to output this cycle I1121 19:31:06.412142 3798194 cpu.go:275] no items to output this cycle E1121 19:31:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:31:16.411980 3798194 memory.go:184] no items to output this cycle I1121 19:31:16.412131 3798194 cpu.go:275] no items to output this cycle E1121 19:31:26.413475 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:31:26.413508 3798194 memory.go:191] Add success. I1121 19:31:26.417717 3798194 cpu.go:282] Add success. I1121 19:31:26.441281 3798194 net.go:642] Add success. I1121 19:31:26.442712 3798194 net.go:778] primary dev: ETH0 I1121 19:31:26.442726 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:31:26.442738 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:31:29.450008 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:31:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:31:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:31:36.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:31:36.412877 3798194 memory.go:184] no items to output this cycle I1121 19:31:36.413019 3798194 cpu.go:275] no items to output this cycle E1121 19:31:46.413011 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:31:46.413035 3798194 memory.go:184] no items to output this cycle I1121 19:31:46.413205 3798194 cpu.go:275] no items to output this cycle E1121 19:31:56.412847 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:31:56.412882 3798194 memory.go:191] Add success. W1121 19:31:56.412930 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:31:56.412941 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:31:56.412945 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:31:56.417269 3798194 cpu.go:282] Add success. I1121 19:31:56.441678 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:31:56.442181 3798194 net.go:642] Add success. I1121 19:31:56.443665 3798194 net.go:778] primary dev: ETH0 I1121 19:31:56.443679 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:31:56.443690 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:31:57.450997 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:31:57.451217 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:31:57.451229 3798194 disk_worker.go:723] disk space is not compliant W1121 19:31:57.451232 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:31:57.460013 3798194 disk_worker.go:509] system disk:vda1 I1121 19:31:57.460054 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:31:58.446400 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:31:59.199143 3798194 disk_info.go:125] begin check local disk info of client I1121 19:31:59.216383 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:31:59.216392 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003be280 0xc0003be2c0] I1121 19:31:59.450140 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:31:59.450216 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:31:59.450241 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:31:59.471135 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:32:06.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:32:06.412877 3798194 memory.go:184] no items to output this cycle I1121 19:32:06.413018 3798194 cpu.go:275] no items to output this cycle E1121 19:32:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:32:16.412257 3798194 memory.go:184] no items to output this cycle I1121 19:32:16.412406 3798194 cpu.go:275] no items to output this cycle E1121 19:32:26.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:32:26.413268 3798194 memory.go:191] Add success. I1121 19:32:26.417579 3798194 cpu.go:282] Add success. I1121 19:32:26.442039 3798194 net.go:642] Add success. I1121 19:32:26.443626 3798194 net.go:778] primary dev: ETH0 I1121 19:32:26.443642 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:32:26.443654 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:32:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:32:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:32:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:32:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:32:36.413259 3798194 memory.go:184] no items to output this cycle I1121 19:32:36.413405 3798194 cpu.go:275] no items to output this cycle E1121 19:32:46.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:32:46.411993 3798194 memory.go:184] no items to output this cycle I1121 19:32:46.412148 3798194 cpu.go:275] no items to output this cycle E1121 19:32:56.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:32:56.412886 3798194 memory.go:191] Add success. W1121 19:32:56.412924 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:32:56.412936 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:32:56.412939 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:32:56.417230 3798194 cpu.go:282] Add success. I1121 19:32:56.441664 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:32:56.442156 3798194 net.go:642] Add success. I1121 19:32:56.443770 3798194 net.go:778] primary dev: ETH0 I1121 19:32:56.443784 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:32:56.443797 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:32:57.451415 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:32:57.451578 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:32:57.451633 3798194 disk_worker.go:723] disk space is not compliant W1121 19:32:57.451636 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:32:57.460029 3798194 disk_worker.go:509] system disk:vda1 I1121 19:32:57.460072 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:32:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:32:59.219146 3798194 disk_info.go:125] begin check local disk info of client I1121 19:32:59.222689 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:32:59.222698 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000426680 0xc0004266c0] I1121 19:32:59.449982 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:32:59.450050 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:32:59.450077 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:32:59.471599 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:33:06.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:33:06.411987 3798194 memory.go:184] no items to output this cycle I1121 19:33:06.412138 3798194 cpu.go:275] no items to output this cycle E1121 19:33:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:33:16.411974 3798194 memory.go:184] no items to output this cycle I1121 19:33:16.412130 3798194 cpu.go:275] no items to output this cycle I1121 19:33:17.545571 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:33:17.545580 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:33:26.411217 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:33:26.411249 3798194 memory.go:191] Add success. I1121 19:33:26.411580 3798194 cpu.go:282] Add success. I1121 19:33:26.442108 3798194 net.go:642] Add success. I1121 19:33:26.446058 3798194 net.go:778] primary dev: ETH0 I1121 19:33:26.446074 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:33:26.446092 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:33:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:33:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:33:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:33:36.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:33:36.413255 3798194 memory.go:184] no items to output this cycle I1121 19:33:36.413398 3798194 cpu.go:275] no items to output this cycle E1121 19:33:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:33:46.411980 3798194 memory.go:184] no items to output this cycle I1121 19:33:46.412139 3798194 cpu.go:275] no items to output this cycle E1121 19:33:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:33:56.411952 3798194 memory.go:191] Add success. W1121 19:33:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:33:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:33:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:33:56.412335 3798194 cpu.go:282] Add success. I1121 19:33:56.441992 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:33:56.443162 3798194 net.go:642] Add success. I1121 19:33:56.447505 3798194 net.go:778] primary dev: ETH0 I1121 19:33:56.447519 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:33:56.447531 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:33:57.450445 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:33:57.450604 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:33:57.450660 3798194 disk_worker.go:723] disk space is not compliant W1121 19:33:57.450663 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:33:57.458774 3798194 disk_worker.go:509] system disk:vda1 I1121 19:33:57.458815 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:33:58.445995 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:33:59.225931 3798194 disk_info.go:125] begin check local disk info of client I1121 19:33:59.229755 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:33:59.229765 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00048e540 0xc00048e580] I1121 19:33:59.449972 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:33:59.450057 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:33:59.450082 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:33:59.471621 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:34:06.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:34:06.411985 3798194 memory.go:184] no items to output this cycle I1121 19:34:06.412146 3798194 cpu.go:275] no items to output this cycle E1121 19:34:16.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:34:16.413271 3798194 memory.go:184] no items to output this cycle I1121 19:34:16.413436 3798194 cpu.go:275] no items to output this cycle E1121 19:34:26.411260 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:34:26.411301 3798194 memory.go:191] Add success. I1121 19:34:26.416198 3798194 cpu.go:282] Add success. I1121 19:34:26.441687 3798194 net.go:642] Add success. I1121 19:34:26.443578 3798194 net.go:778] primary dev: ETH0 I1121 19:34:26.443594 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:34:26.443607 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:34:29.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:34:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:34:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:34:36.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:34:36.412878 3798194 memory.go:184] no items to output this cycle I1121 19:34:36.413028 3798194 cpu.go:275] no items to output this cycle E1121 19:34:46.412334 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:34:46.412358 3798194 memory.go:184] no items to output this cycle I1121 19:34:46.412505 3798194 cpu.go:275] no items to output this cycle E1121 19:34:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:34:56.411952 3798194 memory.go:191] Add success. W1121 19:34:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:34:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:34:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:34:56.412362 3798194 cpu.go:282] Add success. I1121 19:34:56.441952 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:34:56.442563 3798194 net.go:642] Add success. I1121 19:34:56.446491 3798194 net.go:778] primary dev: ETH0 I1121 19:34:56.446505 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:34:56.446519 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:34:57.450703 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:34:57.450916 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:34:57.450926 3798194 disk_worker.go:723] disk space is not compliant W1121 19:34:57.450929 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:34:57.459017 3798194 disk_worker.go:509] system disk:vda1 I1121 19:34:57.459061 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:34:58.445994 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:34:59.229861 3798194 disk_info.go:125] begin check local disk info of client I1121 19:34:59.233720 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:34:59.233729 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037c800 0xc00037c840] I1121 19:34:59.449952 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:34:59.450037 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:34:59.450063 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:34:59.471594 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:35:06.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:35:06.411991 3798194 memory.go:184] no items to output this cycle I1121 19:35:06.412145 3798194 cpu.go:275] no items to output this cycle E1121 19:35:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:35:16.411986 3798194 memory.go:184] no items to output this cycle I1121 19:35:16.412170 3798194 cpu.go:275] no items to output this cycle E1121 19:35:26.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:35:26.412267 3798194 memory.go:191] Add success. I1121 19:35:26.417245 3798194 cpu.go:282] Add success. I1121 19:35:26.441808 3798194 net.go:642] Add success. I1121 19:35:26.443210 3798194 net.go:778] primary dev: ETH0 I1121 19:35:26.443225 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:35:26.443238 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:35:29.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:35:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:35:29.450152 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:35:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:35:36.411977 3798194 memory.go:184] no items to output this cycle I1121 19:35:36.412135 3798194 cpu.go:275] no items to output this cycle E1121 19:35:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:35:46.411977 3798194 memory.go:184] no items to output this cycle I1121 19:35:46.412131 3798194 cpu.go:275] no items to output this cycle E1121 19:35:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:35:56.411953 3798194 memory.go:191] Add success. W1121 19:35:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:35:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:35:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:35:56.412363 3798194 cpu.go:282] Add success. I1121 19:35:56.440884 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:35:56.442389 3798194 event_worker.go:152] Polling the log file for events... I1121 19:35:56.446004 3798194 net.go:642] Add success. I1121 19:35:56.450094 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795259156, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 19:35:56.450230 3798194 net.go:778] primary dev: ETH0 I1121 19:35:56.450245 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:35:56.450258 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:35:56.454274 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 19:35:57.445595 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 19:35:57.445619 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:35:57.445816 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:35:57.445882 3798194 disk_worker.go:723] disk space is not compliant W1121 19:35:57.445885 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:35:57.450809 3798194 disk_worker.go:509] system disk:vda1 I1121 19:35:57.450847 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 19:35:58.448130 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 19:35:58.448146 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:35:59.233834 3798194 disk_info.go:125] begin check local disk info of client I1121 19:35:59.237725 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:35:59.237734 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003aa580 0xc0003aa5c0] E1121 19:35:59.448491 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 19:35:59.450622 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:35:59.450672 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:35:59.450696 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:35:59.471190 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:36:06.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:36:06.411981 3798194 memory.go:184] no items to output this cycle I1121 19:36:06.412151 3798194 cpu.go:275] no items to output this cycle E1121 19:36:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:36:16.411975 3798194 memory.go:184] no items to output this cycle I1121 19:36:16.412131 3798194 cpu.go:275] no items to output this cycle I1121 19:36:17.545724 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:36:17.545732 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:36:26.411221 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:36:26.411253 3798194 memory.go:191] Add success. I1121 19:36:26.411582 3798194 cpu.go:282] Add success. I1121 19:36:26.441301 3798194 net.go:642] Add success. I1121 19:36:26.445594 3798194 net.go:778] primary dev: ETH0 I1121 19:36:26.445611 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:36:26.445623 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:36:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:36:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:36:29.450501 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:36:36.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:36:36.413274 3798194 memory.go:184] no items to output this cycle I1121 19:36:36.413632 3798194 cpu.go:275] no items to output this cycle E1121 19:36:46.411972 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:36:46.411996 3798194 memory.go:184] no items to output this cycle I1121 19:36:46.412162 3798194 cpu.go:275] no items to output this cycle E1121 19:36:56.411930 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:36:56.411963 3798194 memory.go:191] Add success. W1121 19:36:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:36:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:36:56.412007 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:36:56.412387 3798194 cpu.go:282] Add success. I1121 19:36:56.441815 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:36:56.442327 3798194 net.go:642] Add success. I1121 19:36:56.446375 3798194 net.go:778] primary dev: ETH0 I1121 19:36:56.446390 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:36:56.446402 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:36:57.450329 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:36:57.450487 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:36:57.450540 3798194 disk_worker.go:723] disk space is not compliant W1121 19:36:57.450543 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:36:57.458614 3798194 disk_worker.go:509] system disk:vda1 I1121 19:36:57.458659 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:36:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:36:59.241381 3798194 disk_info.go:125] begin check local disk info of client I1121 19:36:59.245235 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:36:59.245244 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047d4c0 0xc00047d500] I1121 19:36:59.450008 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:36:59.450076 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:36:59.450100 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:36:59.471619 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:37:06.413191 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:37:06.413220 3798194 memory.go:184] no items to output this cycle I1121 19:37:06.413380 3798194 cpu.go:275] no items to output this cycle E1121 19:37:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:37:16.412257 3798194 memory.go:184] no items to output this cycle I1121 19:37:16.412401 3798194 cpu.go:275] no items to output this cycle E1121 19:37:26.413249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:37:26.413294 3798194 memory.go:191] Add success. I1121 19:37:26.417812 3798194 cpu.go:282] Add success. I1121 19:37:26.441641 3798194 net.go:642] Add success. I1121 19:37:26.443337 3798194 net.go:778] primary dev: ETH0 I1121 19:37:26.443351 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:37:26.443365 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:37:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:37:29.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:37:29.450512 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:37:36.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:37:36.411987 3798194 memory.go:184] no items to output this cycle I1121 19:37:36.412143 3798194 cpu.go:275] no items to output this cycle E1121 19:37:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:37:46.411981 3798194 memory.go:184] no items to output this cycle I1121 19:37:46.412149 3798194 cpu.go:275] no items to output this cycle E1121 19:37:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:37:56.411952 3798194 memory.go:191] Add success. W1121 19:37:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:37:56.412011 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:37:56.412016 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:37:56.412411 3798194 cpu.go:282] Add success. I1121 19:37:56.440967 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:37:56.441568 3798194 net.go:642] Add success. I1121 19:37:56.445624 3798194 net.go:778] primary dev: ETH0 I1121 19:37:56.445639 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:37:56.445652 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:37:57.451150 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:37:57.451501 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:37:57.451559 3798194 disk_worker.go:723] disk space is not compliant W1121 19:37:57.451562 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:37:57.460812 3798194 disk_worker.go:509] system disk:vda1 I1121 19:37:57.460852 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:37:58.446361 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:37:59.248142 3798194 disk_info.go:125] begin check local disk info of client I1121 19:37:59.251668 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:37:59.251677 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004798c0 0xc000479900] I1121 19:37:59.449665 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:37:59.449734 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:37:59.449758 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:37:59.471182 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:38:06.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:38:06.412261 3798194 memory.go:184] no items to output this cycle I1121 19:38:06.412400 3798194 cpu.go:275] no items to output this cycle E1121 19:38:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:38:16.411983 3798194 memory.go:184] no items to output this cycle I1121 19:38:16.412143 3798194 cpu.go:275] no items to output this cycle E1121 19:38:26.411930 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:38:26.411961 3798194 memory.go:191] Add success. I1121 19:38:26.412306 3798194 cpu.go:282] Add success. I1121 19:38:26.441977 3798194 net.go:642] Add success. I1121 19:38:26.446009 3798194 net.go:778] primary dev: ETH0 I1121 19:38:26.446026 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:38:26.446040 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:38:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:38:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:38:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:38:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:38:36.412257 3798194 memory.go:184] no items to output this cycle I1121 19:38:36.412397 3798194 cpu.go:275] no items to output this cycle E1121 19:38:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:38:46.411976 3798194 memory.go:184] no items to output this cycle I1121 19:38:46.412139 3798194 cpu.go:275] no items to output this cycle E1121 19:38:56.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:38:56.413267 3798194 memory.go:191] Add success. W1121 19:38:56.413300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:38:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:38:56.413315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:38:56.417357 3798194 cpu.go:282] Add success. I1121 19:38:56.441737 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:38:56.442229 3798194 net.go:642] Add success. I1121 19:38:56.443791 3798194 net.go:778] primary dev: ETH0 I1121 19:38:56.443808 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:38:56.443820 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:38:57.450986 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:38:57.451230 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:38:57.451241 3798194 disk_worker.go:723] disk space is not compliant W1121 19:38:57.451244 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:38:57.465030 3798194 disk_worker.go:509] system disk:vda1 I1121 19:38:57.465075 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:38:58.445984 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:38:59.251773 3798194 disk_info.go:125] begin check local disk info of client I1121 19:38:59.255997 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:38:59.256006 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ceb00 0xc0003ceb40] I1121 19:38:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:38:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:38:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:38:59.471643 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:39:06.411242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:39:06.411264 3798194 memory.go:184] no items to output this cycle I1121 19:39:06.411409 3798194 cpu.go:275] no items to output this cycle E1121 19:39:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:39:16.411984 3798194 memory.go:184] no items to output this cycle I1121 19:39:16.412144 3798194 cpu.go:275] no items to output this cycle I1121 19:39:17.546592 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:39:17.546600 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:39:26.412868 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:39:26.412913 3798194 memory.go:191] Add success. I1121 19:39:26.417358 3798194 cpu.go:282] Add success. I1121 19:39:26.441864 3798194 net.go:642] Add success. I1121 19:39:26.443269 3798194 net.go:778] primary dev: ETH0 I1121 19:39:26.443283 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:39:26.443296 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:39:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:39:29.450467 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:39:29.450496 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:39:36.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:39:36.411980 3798194 memory.go:184] no items to output this cycle I1121 19:39:36.412132 3798194 cpu.go:275] no items to output this cycle E1121 19:39:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:39:46.411981 3798194 memory.go:184] no items to output this cycle I1121 19:39:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 19:39:56.411938 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:39:56.411973 3798194 memory.go:191] Add success. W1121 19:39:56.412001 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:39:56.412013 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:39:56.412016 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:39:56.412378 3798194 cpu.go:282] Add success. I1121 19:39:56.441880 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:39:56.442406 3798194 net.go:642] Add success. I1121 19:39:56.446748 3798194 net.go:778] primary dev: ETH0 I1121 19:39:56.446763 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:39:56.446777 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:39:57.451781 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:39:57.451946 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:39:57.451997 3798194 disk_worker.go:723] disk space is not compliant W1121 19:39:57.452000 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:39:57.460342 3798194 disk_worker.go:509] system disk:vda1 I1121 19:39:57.460386 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:39:58.446013 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:39:59.258145 3798194 disk_info.go:125] begin check local disk info of client I1121 19:39:59.261967 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:39:59.261976 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000322b80 0xc000322bc0] I1121 19:39:59.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:39:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:39:59.450140 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:39:59.471704 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:40:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:40:06.412255 3798194 memory.go:184] no items to output this cycle I1121 19:40:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 19:40:16.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:40:16.412258 3798194 memory.go:184] no items to output this cycle I1121 19:40:16.412405 3798194 cpu.go:275] no items to output this cycle E1121 19:40:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:40:26.411957 3798194 memory.go:191] Add success. I1121 19:40:26.412315 3798194 cpu.go:282] Add success. I1121 19:40:26.441756 3798194 net.go:642] Add success. I1121 19:40:26.445891 3798194 net.go:778] primary dev: ETH0 I1121 19:40:26.445906 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:40:26.445923 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:40:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:40:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:40:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:40:36.413160 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:40:36.413183 3798194 memory.go:184] no items to output this cycle I1121 19:40:36.413337 3798194 cpu.go:275] no items to output this cycle E1121 19:40:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:40:46.411978 3798194 memory.go:184] no items to output this cycle I1121 19:40:46.412148 3798194 cpu.go:275] no items to output this cycle E1121 19:40:56.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:40:56.412895 3798194 memory.go:191] Add success. W1121 19:40:56.412923 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:40:56.412935 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:40:56.412939 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:40:56.417277 3798194 cpu.go:282] Add success. I1121 19:40:56.444564 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:40:56.445045 3798194 net.go:642] Add success. I1121 19:40:56.446581 3798194 net.go:778] primary dev: ETH0 I1121 19:40:56.446596 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:40:56.446608 3798194 net.go:659] Add eni metrics success, count: 7 E1121 19:40:57.450479 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 19:40:57.450493 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:40:57.450640 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:40:57.450691 3798194 disk_worker.go:723] disk space is not compliant W1121 19:40:57.450694 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:40:57.458762 3798194 disk_worker.go:509] system disk:vda1 I1121 19:40:57.458804 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 19:40:58.448347 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 19:40:58.448362 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:40:59.262088 3798194 disk_info.go:125] begin check local disk info of client I1121 19:40:59.265615 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:40:59.265624 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e05c0 0xc0002e0600] I1121 19:40:59.449803 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:40:59.449853 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:40:59.449885 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:40:59.449992 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 19:40:59.471479 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:41:06.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:41:06.411988 3798194 memory.go:184] no items to output this cycle I1121 19:41:06.412149 3798194 cpu.go:275] no items to output this cycle E1121 19:41:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:41:16.412258 3798194 memory.go:184] no items to output this cycle I1121 19:41:16.412416 3798194 cpu.go:275] no items to output this cycle E1121 19:41:26.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:41:26.411950 3798194 memory.go:191] Add success. I1121 19:41:26.412297 3798194 cpu.go:282] Add success. I1121 19:41:26.442023 3798194 net.go:642] Add success. I1121 19:41:26.446137 3798194 net.go:778] primary dev: ETH0 I1121 19:41:26.446152 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:41:26.446165 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:41:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:41:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:41:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:41:36.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:41:36.412883 3798194 memory.go:184] no items to output this cycle I1121 19:41:36.413026 3798194 cpu.go:275] no items to output this cycle E1121 19:41:46.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:41:46.412252 3798194 memory.go:184] no items to output this cycle I1121 19:41:46.412397 3798194 cpu.go:275] no items to output this cycle E1121 19:41:56.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:41:56.413269 3798194 memory.go:191] Add success. W1121 19:41:56.413298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:41:56.413309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:41:56.413312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:41:56.418895 3798194 cpu.go:282] Add success. I1121 19:41:56.441327 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:41:56.441890 3798194 net.go:642] Add success. I1121 19:41:56.443464 3798194 net.go:778] primary dev: ETH0 I1121 19:41:56.443479 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:41:56.443493 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:41:57.446426 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:41:57.446682 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:41:57.446694 3798194 disk_worker.go:723] disk space is not compliant W1121 19:41:57.446697 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:41:57.449491 3798194 disk_worker.go:509] system disk:vda1 I1121 19:41:57.449530 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:41:58.446161 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:41:59.265730 3798194 disk_info.go:125] begin check local disk info of client I1121 19:41:59.269606 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:41:59.269616 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041c2c0 0xc00041c300] I1121 19:41:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:41:59.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:41:59.450128 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:41:59.471638 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:42:06.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:42:06.412264 3798194 memory.go:184] no items to output this cycle I1121 19:42:06.412415 3798194 cpu.go:275] no items to output this cycle E1121 19:42:16.412954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:42:16.412978 3798194 memory.go:184] no items to output this cycle I1121 19:42:16.413142 3798194 cpu.go:275] no items to output this cycle I1121 19:42:17.547272 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:42:17.547280 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:42:26.411384 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:42:26.411418 3798194 memory.go:191] Add success. I1121 19:42:26.411762 3798194 cpu.go:282] Add success. I1121 19:42:26.441351 3798194 net.go:642] Add success. I1121 19:42:26.445281 3798194 net.go:778] primary dev: ETH0 I1121 19:42:26.445297 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:42:26.445309 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:42:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:42:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:42:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:42:36.412422 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:42:36.412446 3798194 memory.go:184] no items to output this cycle I1121 19:42:36.412602 3798194 cpu.go:275] no items to output this cycle E1121 19:42:46.411942 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:42:46.411964 3798194 memory.go:184] no items to output this cycle I1121 19:42:46.412130 3798194 cpu.go:275] no items to output this cycle E1121 19:42:56.412650 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:42:56.412683 3798194 memory.go:191] Add success. W1121 19:42:56.412718 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:42:56.412730 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:42:56.412733 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:42:56.417550 3798194 cpu.go:282] Add success. I1121 19:42:56.440931 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:42:56.441530 3798194 net.go:642] Add success. I1121 19:42:56.442956 3798194 net.go:778] primary dev: ETH0 I1121 19:42:56.442970 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:42:56.442983 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:42:57.445775 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:42:57.446047 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:42:57.446058 3798194 disk_worker.go:723] disk space is not compliant W1121 19:42:57.446061 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:42:57.448192 3798194 disk_worker.go:509] system disk:vda1 I1121 19:42:57.448234 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:42:58.445992 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:42:59.269713 3798194 disk_info.go:125] begin check local disk info of client I1121 19:42:59.273739 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:42:59.273749 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003976c0 0xc000397700] I1121 19:42:59.449645 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:42:59.449717 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:42:59.449741 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:42:59.471186 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:43:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:43:06.411978 3798194 memory.go:184] no items to output this cycle I1121 19:43:06.412134 3798194 cpu.go:275] no items to output this cycle E1121 19:43:16.412250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:43:16.412272 3798194 memory.go:184] no items to output this cycle I1121 19:43:16.412416 3798194 cpu.go:275] no items to output this cycle E1121 19:43:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:43:26.411956 3798194 memory.go:191] Add success. I1121 19:43:26.412305 3798194 cpu.go:282] Add success. I1121 19:43:26.441886 3798194 net.go:642] Add success. I1121 19:43:26.446240 3798194 net.go:778] primary dev: ETH0 I1121 19:43:26.446257 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:43:26.446277 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:43:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:43:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:43:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:43:36.412091 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:43:36.412132 3798194 memory.go:184] no items to output this cycle I1121 19:43:36.412281 3798194 cpu.go:275] no items to output this cycle E1121 19:43:46.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:43:46.412262 3798194 memory.go:184] no items to output this cycle I1121 19:43:46.412413 3798194 cpu.go:275] no items to output this cycle E1121 19:43:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:43:56.411952 3798194 memory.go:191] Add success. W1121 19:43:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:43:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:43:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:43:56.412349 3798194 cpu.go:282] Add success. I1121 19:43:56.441825 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:43:56.442419 3798194 net.go:642] Add success. I1121 19:43:56.446596 3798194 net.go:778] primary dev: ETH0 I1121 19:43:56.446611 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:43:56.446624 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:43:57.450445 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:43:57.450610 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:43:57.450666 3798194 disk_worker.go:723] disk space is not compliant W1121 19:43:57.450670 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:43:57.460350 3798194 disk_worker.go:509] system disk:vda1 I1121 19:43:57.460392 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:43:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:43:59.273852 3798194 disk_info.go:125] begin check local disk info of client I1121 19:43:59.278002 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:43:59.278011 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003dbb00 0xc0003dbb40] I1121 19:43:59.450472 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:43:59.450552 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:43:59.450578 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:43:59.471099 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:44:06.412097 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:44:06.412137 3798194 memory.go:184] no items to output this cycle I1121 19:44:06.412297 3798194 cpu.go:275] no items to output this cycle E1121 19:44:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:44:16.411976 3798194 memory.go:184] no items to output this cycle I1121 19:44:16.412132 3798194 cpu.go:275] no items to output this cycle E1121 19:44:26.413314 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:44:26.413364 3798194 memory.go:191] Add success. I1121 19:44:26.418144 3798194 cpu.go:282] Add success. I1121 19:44:26.442039 3798194 net.go:642] Add success. I1121 19:44:26.443690 3798194 net.go:778] primary dev: ETH0 I1121 19:44:26.443705 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:44:26.443719 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:44:29.450003 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:44:29.450084 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:44:29.450123 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:44:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:44:36.412256 3798194 memory.go:184] no items to output this cycle I1121 19:44:36.412398 3798194 cpu.go:275] no items to output this cycle E1121 19:44:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:44:46.411982 3798194 memory.go:184] no items to output this cycle I1121 19:44:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 19:44:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:44:56.411953 3798194 memory.go:191] Add success. W1121 19:44:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:44:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:44:56.412000 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:44:56.412341 3798194 cpu.go:282] Add success. I1121 19:44:56.441906 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:44:56.442945 3798194 net.go:642] Add success. I1121 19:44:56.446957 3798194 net.go:778] primary dev: ETH0 I1121 19:44:56.446972 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:44:56.446985 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:44:57.450698 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:44:57.450863 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:44:57.450919 3798194 disk_worker.go:723] disk space is not compliant W1121 19:44:57.450922 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:44:57.459076 3798194 disk_worker.go:509] system disk:vda1 I1121 19:44:57.459154 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:44:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:44:59.278132 3798194 disk_info.go:125] begin check local disk info of client I1121 19:44:59.281968 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:44:59.281976 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004af8c0 0xc0004af900] I1121 19:44:59.450469 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:44:59.450541 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:44:59.450569 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:44:59.471078 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:45:06.412771 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:45:06.412794 3798194 memory.go:184] no items to output this cycle I1121 19:45:06.412946 3798194 cpu.go:275] no items to output this cycle E1121 19:45:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:45:16.411980 3798194 memory.go:184] no items to output this cycle I1121 19:45:16.412153 3798194 cpu.go:275] no items to output this cycle I1121 19:45:17.547446 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:45:17.547455 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:45:26.412499 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:45:26.412535 3798194 memory.go:191] Add success. I1121 19:45:26.417333 3798194 cpu.go:282] Add success. I1121 19:45:26.441884 3798194 net.go:642] Add success. I1121 19:45:26.443503 3798194 net.go:778] primary dev: ETH0 I1121 19:45:26.443519 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:45:26.443531 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:45:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:45:29.450128 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:45:29.450156 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:45:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:45:36.413261 3798194 memory.go:184] no items to output this cycle I1121 19:45:36.413403 3798194 cpu.go:275] no items to output this cycle E1121 19:45:46.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:45:46.412257 3798194 memory.go:184] no items to output this cycle I1121 19:45:46.412408 3798194 cpu.go:275] no items to output this cycle E1121 19:45:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:45:56.411947 3798194 memory.go:191] Add success. W1121 19:45:56.411975 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:45:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:45:56.411990 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:45:56.412344 3798194 cpu.go:282] Add success. I1121 19:45:56.441654 3798194 event_worker.go:152] Polling the log file for events... I1121 19:45:56.446421 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:45:56.447010 3798194 net.go:642] Add success. I1121 19:45:56.450848 3798194 net.go:778] primary dev: ETH0 I1121 19:45:56.450863 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:45:56.450874 3798194 net.go:659] Add eni metrics success, count: 7 E1121 19:45:57.445486 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 19:45:57.445500 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:45:57.445752 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:45:57.445762 3798194 disk_worker.go:723] disk space is not compliant W1121 19:45:57.445765 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:45:57.447832 3798194 disk_worker.go:509] system disk:vda1 I1121 19:45:57.447873 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 19:45:58.448409 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 19:45:58.448424 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:45:59.282075 3798194 disk_info.go:125] begin check local disk info of client I1121 19:45:59.285627 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:45:59.285636 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f48c0 0xc0003f4900] E1121 19:45:59.448534 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 19:45:59.449606 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:45:59.449655 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:45:59.449677 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:45:59.471188 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:46:06.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:46:06.412249 3798194 memory.go:184] no items to output this cycle I1121 19:46:06.412393 3798194 cpu.go:275] no items to output this cycle E1121 19:46:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:46:16.411985 3798194 memory.go:184] no items to output this cycle I1121 19:46:16.412143 3798194 cpu.go:275] no items to output this cycle E1121 19:46:26.411932 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:46:26.411965 3798194 memory.go:191] Add success. I1121 19:46:26.412336 3798194 cpu.go:282] Add success. I1121 19:46:26.441950 3798194 net.go:642] Add success. I1121 19:46:26.454229 3798194 net.go:778] primary dev: ETH0 I1121 19:46:26.454253 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:46:26.454272 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:46:29.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:46:29.450467 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:46:29.450495 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:46:36.412970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:46:36.412994 3798194 memory.go:184] no items to output this cycle I1121 19:46:36.413166 3798194 cpu.go:275] no items to output this cycle E1121 19:46:46.413046 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:46:46.413069 3798194 memory.go:184] no items to output this cycle I1121 19:46:46.413238 3798194 cpu.go:275] no items to output this cycle E1121 19:46:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:46:56.411950 3798194 memory.go:191] Add success. W1121 19:46:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:46:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:46:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:46:56.412345 3798194 cpu.go:282] Add success. I1121 19:46:56.441816 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:46:56.442343 3798194 net.go:642] Add success. I1121 19:46:56.446392 3798194 net.go:778] primary dev: ETH0 I1121 19:46:56.446408 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:46:56.446421 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:46:57.450610 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:46:57.450805 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:46:57.450816 3798194 disk_worker.go:723] disk space is not compliant W1121 19:46:57.450819 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:46:57.459230 3798194 disk_worker.go:509] system disk:vda1 I1121 19:46:57.459270 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:46:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:46:59.285792 3798194 disk_info.go:125] begin check local disk info of client I1121 19:46:59.289677 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:46:59.289685 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000388240 0xc000388280] I1121 19:46:59.450631 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:46:59.450707 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:46:59.450732 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:46:59.471260 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:47:06.412766 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:47:06.412787 3798194 memory.go:184] no items to output this cycle I1121 19:47:06.412938 3798194 cpu.go:275] no items to output this cycle E1121 19:47:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:47:16.411978 3798194 memory.go:184] no items to output this cycle I1121 19:47:16.412137 3798194 cpu.go:275] no items to output this cycle E1121 19:47:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:47:26.413265 3798194 memory.go:191] Add success. I1121 19:47:26.426341 3798194 cpu.go:282] Add success. I1121 19:47:26.442049 3798194 net.go:642] Add success. I1121 19:47:26.443722 3798194 net.go:778] primary dev: ETH0 I1121 19:47:26.443737 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:47:26.443748 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:47:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:47:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:47:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:47:36.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:47:36.411983 3798194 memory.go:184] no items to output this cycle I1121 19:47:36.412139 3798194 cpu.go:275] no items to output this cycle E1121 19:47:46.413322 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:47:46.413348 3798194 memory.go:184] no items to output this cycle I1121 19:47:46.413498 3798194 cpu.go:275] no items to output this cycle E1121 19:47:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:47:56.412266 3798194 memory.go:191] Add success. W1121 19:47:56.412292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:47:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:47:56.412306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:47:56.416482 3798194 cpu.go:282] Add success. I1121 19:47:56.441913 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:47:56.442488 3798194 net.go:642] Add success. I1121 19:47:56.444097 3798194 net.go:778] primary dev: ETH0 I1121 19:47:56.444121 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:47:56.444135 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:47:57.451059 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:47:57.451234 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:47:57.451288 3798194 disk_worker.go:723] disk space is not compliant W1121 19:47:57.451291 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:47:57.459872 3798194 disk_worker.go:509] system disk:vda1 I1121 19:47:57.459915 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:47:58.446233 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:47:59.289792 3798194 disk_info.go:125] begin check local disk info of client I1121 19:47:59.293637 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:47:59.293649 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000440440 0xc000440480] I1121 19:47:59.450582 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:47:59.450659 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:47:59.450683 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:47:59.471191 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:48:06.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:48:06.412259 3798194 memory.go:184] no items to output this cycle I1121 19:48:06.412405 3798194 cpu.go:275] no items to output this cycle E1121 19:48:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:48:16.411974 3798194 memory.go:184] no items to output this cycle I1121 19:48:16.412130 3798194 cpu.go:275] no items to output this cycle I1121 19:48:17.548269 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:48:17.548278 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:48:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:48:26.413263 3798194 memory.go:191] Add success. I1121 19:48:26.417659 3798194 cpu.go:282] Add success. I1121 19:48:26.442192 3798194 net.go:642] Add success. I1121 19:48:26.443628 3798194 net.go:778] primary dev: ETH0 I1121 19:48:26.443641 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:48:26.443655 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:48:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:48:29.450465 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:48:29.450490 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:48:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:48:36.411978 3798194 memory.go:184] no items to output this cycle I1121 19:48:36.412130 3798194 cpu.go:275] no items to output this cycle E1121 19:48:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:48:46.411976 3798194 memory.go:184] no items to output this cycle I1121 19:48:46.412133 3798194 cpu.go:275] no items to output this cycle E1121 19:48:56.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:48:56.413261 3798194 memory.go:191] Add success. W1121 19:48:56.413295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:48:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:48:56.413309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:48:56.417821 3798194 cpu.go:282] Add success. I1121 19:48:56.441230 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:48:56.441708 3798194 net.go:642] Add success. I1121 19:48:56.443237 3798194 net.go:778] primary dev: ETH0 I1121 19:48:56.443252 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:48:56.443265 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:48:57.450970 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:48:57.451174 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:48:57.451232 3798194 disk_worker.go:723] disk space is not compliant W1121 19:48:57.451236 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:48:57.460386 3798194 disk_worker.go:509] system disk:vda1 I1121 19:48:57.460425 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:48:58.446369 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:48:59.293748 3798194 disk_info.go:125] begin check local disk info of client I1121 19:48:59.297610 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:48:59.297619 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00049f400 0xc00049f440] I1121 19:48:59.450409 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:48:59.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:48:59.450509 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:48:59.471026 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:49:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:49:06.412260 3798194 memory.go:184] no items to output this cycle I1121 19:49:06.412403 3798194 cpu.go:275] no items to output this cycle E1121 19:49:16.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:49:16.413249 3798194 memory.go:184] no items to output this cycle I1121 19:49:16.413393 3798194 cpu.go:275] no items to output this cycle E1121 19:49:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:49:26.413266 3798194 memory.go:191] Add success. I1121 19:49:26.417602 3798194 cpu.go:282] Add success. I1121 19:49:26.442152 3798194 net.go:642] Add success. I1121 19:49:26.443778 3798194 net.go:778] primary dev: ETH0 I1121 19:49:26.443792 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:49:26.443805 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:49:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:49:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:49:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:49:36.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:49:36.412882 3798194 memory.go:184] no items to output this cycle I1121 19:49:36.413024 3798194 cpu.go:275] no items to output this cycle I1121 19:49:46.411084 3798194 cpu.go:275] no items to output this cycle E1121 19:49:46.411177 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:49:46.411194 3798194 memory.go:184] no items to output this cycle E1121 19:49:56.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:49:56.413264 3798194 memory.go:191] Add success. W1121 19:49:56.413291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:49:56.413302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:49:56.413305 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:49:56.418483 3798194 cpu.go:282] Add success. I1121 19:49:56.441870 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:49:56.442371 3798194 net.go:642] Add success. I1121 19:49:56.444218 3798194 net.go:778] primary dev: ETH0 I1121 19:49:56.444233 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:49:56.444244 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:49:57.449151 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:49:57.449338 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:49:57.449394 3798194 disk_worker.go:723] disk space is not compliant W1121 19:49:57.449397 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:49:57.451265 3798194 disk_worker.go:509] system disk:vda1 I1121 19:49:57.451304 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:49:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:49:59.297706 3798194 disk_info.go:125] begin check local disk info of client I1121 19:49:59.301558 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:49:59.301568 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f2200 0xc0003f2240] I1121 19:49:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:49:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:49:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:49:59.471653 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:50:06.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:50:06.411990 3798194 memory.go:184] no items to output this cycle I1121 19:50:06.412151 3798194 cpu.go:275] no items to output this cycle E1121 19:50:16.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:50:16.412257 3798194 memory.go:184] no items to output this cycle I1121 19:50:16.412416 3798194 cpu.go:275] no items to output this cycle E1121 19:50:26.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:50:26.413269 3798194 memory.go:191] Add success. I1121 19:50:26.417434 3798194 cpu.go:282] Add success. I1121 19:50:26.442097 3798194 net.go:642] Add success. I1121 19:50:26.443668 3798194 net.go:778] primary dev: ETH0 I1121 19:50:26.443684 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:50:26.443714 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:50:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:50:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:50:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:50:36.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:50:36.412878 3798194 memory.go:184] no items to output this cycle I1121 19:50:36.413023 3798194 cpu.go:275] no items to output this cycle E1121 19:50:46.411934 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:50:46.411957 3798194 memory.go:184] no items to output this cycle I1121 19:50:46.412097 3798194 cpu.go:275] no items to output this cycle E1121 19:50:56.411909 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:50:56.411940 3798194 memory.go:191] Add success. W1121 19:50:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:50:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:50:56.411996 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:50:56.412342 3798194 cpu.go:282] Add success. I1121 19:50:56.445049 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:50:56.445540 3798194 net.go:642] Add success. I1121 19:50:56.449913 3798194 net.go:778] primary dev: ETH0 I1121 19:50:56.449928 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:50:56.449944 3798194 net.go:659] Add eni metrics success, count: 7 E1121 19:50:57.450631 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 19:50:57.450644 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:50:57.450844 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:50:57.450855 3798194 disk_worker.go:723] disk space is not compliant W1121 19:50:57.450858 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:50:57.459235 3798194 disk_worker.go:509] system disk:vda1 I1121 19:50:57.459280 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 19:50:58.448085 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 19:50:58.448099 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:50:59.305145 3798194 disk_info.go:125] begin check local disk info of client I1121 19:50:59.308976 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:50:59.308985 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000484740 0xc000484780] E1121 19:50:59.448215 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 19:50:59.450344 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:50:59.450395 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:50:59.450414 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:50:59.470919 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:51:06.412092 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:51:06.412131 3798194 memory.go:184] no items to output this cycle I1121 19:51:06.412285 3798194 cpu.go:275] no items to output this cycle E1121 19:51:16.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:51:16.411989 3798194 memory.go:184] no items to output this cycle I1121 19:51:16.412149 3798194 cpu.go:275] no items to output this cycle I1121 19:51:17.548435 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:51:17.548444 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:51:26.412484 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:51:26.412522 3798194 memory.go:191] Add success. I1121 19:51:26.416750 3798194 cpu.go:282] Add success. I1121 19:51:26.442178 3798194 net.go:642] Add success. I1121 19:51:26.443752 3798194 net.go:778] primary dev: ETH0 I1121 19:51:26.443767 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:51:26.443779 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:51:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:51:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:51:29.450496 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:51:36.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:51:36.411983 3798194 memory.go:184] no items to output this cycle I1121 19:51:36.412138 3798194 cpu.go:275] no items to output this cycle E1121 19:51:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:51:46.411991 3798194 memory.go:184] no items to output this cycle I1121 19:51:46.412152 3798194 cpu.go:275] no items to output this cycle E1121 19:51:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:51:56.411950 3798194 memory.go:191] Add success. W1121 19:51:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:51:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:51:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:51:56.412346 3798194 cpu.go:282] Add success. I1121 19:51:56.441948 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:51:56.442843 3798194 net.go:642] Add success. I1121 19:51:56.447083 3798194 net.go:778] primary dev: ETH0 I1121 19:51:56.447099 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:51:56.447135 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:51:57.450646 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:51:57.450806 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:51:57.450859 3798194 disk_worker.go:723] disk space is not compliant W1121 19:51:57.450862 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:51:57.459255 3798194 disk_worker.go:509] system disk:vda1 I1121 19:51:57.459295 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:51:58.446377 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:51:59.312730 3798194 disk_info.go:125] begin check local disk info of client I1121 19:51:59.316560 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:51:59.316569 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004e1a40 0xc0004e1a80] I1121 19:51:59.450006 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:51:59.450073 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:51:59.450096 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:51:59.471637 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:52:06.412021 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:52:06.412045 3798194 memory.go:184] no items to output this cycle I1121 19:52:06.412204 3798194 cpu.go:275] no items to output this cycle E1121 19:52:16.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:52:16.413261 3798194 memory.go:184] no items to output this cycle I1121 19:52:16.413412 3798194 cpu.go:275] no items to output this cycle E1121 19:52:26.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:52:26.413279 3798194 memory.go:191] Add success. I1121 19:52:26.417619 3798194 cpu.go:282] Add success. I1121 19:52:26.442192 3798194 net.go:642] Add success. I1121 19:52:26.443795 3798194 net.go:778] primary dev: ETH0 I1121 19:52:26.443811 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:52:26.443825 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:52:29.449612 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:52:29.449690 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:52:29.449715 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:52:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:52:36.412253 3798194 memory.go:184] no items to output this cycle I1121 19:52:36.412396 3798194 cpu.go:275] no items to output this cycle E1121 19:52:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:52:46.413258 3798194 memory.go:184] no items to output this cycle I1121 19:52:46.413417 3798194 cpu.go:275] no items to output this cycle E1121 19:52:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:52:56.413265 3798194 memory.go:191] Add success. W1121 19:52:56.413294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:52:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:52:56.413309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:52:56.417798 3798194 cpu.go:282] Add success. I1121 19:52:56.441327 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:52:56.441801 3798194 net.go:642] Add success. I1121 19:52:56.443423 3798194 net.go:778] primary dev: ETH0 I1121 19:52:56.443438 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:52:56.443450 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:52:57.451314 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:52:57.451524 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:52:57.451534 3798194 disk_worker.go:723] disk space is not compliant W1121 19:52:57.451537 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:52:57.460051 3798194 disk_worker.go:509] system disk:vda1 I1121 19:52:57.460098 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:52:58.446390 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:52:59.316668 3798194 disk_info.go:125] begin check local disk info of client I1121 19:52:59.320460 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:52:59.320469 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040c780 0xc00040c7c0] I1121 19:52:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:52:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:52:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:52:59.471691 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:53:06.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:53:06.412263 3798194 memory.go:184] no items to output this cycle I1121 19:53:06.412405 3798194 cpu.go:275] no items to output this cycle E1121 19:53:16.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:53:16.413257 3798194 memory.go:184] no items to output this cycle I1121 19:53:16.413404 3798194 cpu.go:275] no items to output this cycle E1121 19:53:26.411819 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:53:26.411863 3798194 memory.go:191] Add success. I1121 19:53:26.416504 3798194 cpu.go:282] Add success. I1121 19:53:26.442094 3798194 net.go:642] Add success. I1121 19:53:26.443733 3798194 net.go:778] primary dev: ETH0 I1121 19:53:26.443748 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:53:26.443761 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:53:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:53:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:53:29.450502 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:53:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:53:36.411982 3798194 memory.go:184] no items to output this cycle I1121 19:53:36.412133 3798194 cpu.go:275] no items to output this cycle E1121 19:53:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:53:46.411975 3798194 memory.go:184] no items to output this cycle I1121 19:53:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 19:53:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:53:56.411950 3798194 memory.go:191] Add success. W1121 19:53:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:53:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:53:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:53:56.412361 3798194 cpu.go:282] Add success. I1121 19:53:56.441835 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:53:56.442499 3798194 net.go:642] Add success. I1121 19:53:56.446446 3798194 net.go:778] primary dev: ETH0 I1121 19:53:56.446461 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:53:56.446473 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:53:57.450767 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:53:57.450975 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:53:57.450985 3798194 disk_worker.go:723] disk space is not compliant W1121 19:53:57.450989 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:53:57.459782 3798194 disk_worker.go:509] system disk:vda1 I1121 19:53:57.459827 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:53:58.447170 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:53:59.324145 3798194 disk_info.go:125] begin check local disk info of client I1121 19:53:59.328013 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:53:59.328022 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030d340 0xc00030d380] I1121 19:53:59.449666 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:53:59.449746 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:53:59.449771 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:53:59.471337 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:54:06.412981 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:54:06.413007 3798194 memory.go:184] no items to output this cycle I1121 19:54:06.413166 3798194 cpu.go:275] no items to output this cycle E1121 19:54:16.411973 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:54:16.411996 3798194 memory.go:184] no items to output this cycle I1121 19:54:16.412166 3798194 cpu.go:275] no items to output this cycle I1121 19:54:17.548605 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:54:17.548615 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:54:26.413547 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:54:26.413581 3798194 memory.go:191] Add success. I1121 19:54:26.418069 3798194 cpu.go:282] Add success. I1121 19:54:26.441901 3798194 net.go:642] Add success. I1121 19:54:26.443553 3798194 net.go:778] primary dev: ETH0 I1121 19:54:26.443569 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:54:26.443582 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:54:29.450626 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:54:29.450697 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:54:29.450723 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:54:36.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:54:36.413264 3798194 memory.go:184] no items to output this cycle I1121 19:54:36.413402 3798194 cpu.go:275] no items to output this cycle E1121 19:54:46.412743 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:54:46.412768 3798194 memory.go:184] no items to output this cycle I1121 19:54:46.412912 3798194 cpu.go:275] no items to output this cycle E1121 19:54:56.413408 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:54:56.413457 3798194 memory.go:191] Add success. W1121 19:54:56.413494 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:54:56.413507 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:54:56.413511 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:54:56.418475 3798194 cpu.go:282] Add success. I1121 19:54:56.441091 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:54:56.441694 3798194 net.go:642] Add success. I1121 19:54:56.443262 3798194 net.go:778] primary dev: ETH0 I1121 19:54:56.443277 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:54:56.443290 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:54:57.446482 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:54:57.446696 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:54:57.446763 3798194 disk_worker.go:723] disk space is not compliant W1121 19:54:57.446767 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:54:57.449570 3798194 disk_worker.go:509] system disk:vda1 I1121 19:54:57.449610 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:54:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:54:59.330734 3798194 disk_info.go:125] begin check local disk info of client I1121 19:54:59.334556 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:54:59.334565 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ebc0 0xc00030ec00] I1121 19:54:59.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:54:59.450071 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:54:59.450094 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:54:59.471613 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:55:06.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:55:06.413260 3798194 memory.go:184] no items to output this cycle I1121 19:55:06.413406 3798194 cpu.go:275] no items to output this cycle E1121 19:55:16.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:55:16.413262 3798194 memory.go:184] no items to output this cycle I1121 19:55:16.413411 3798194 cpu.go:275] no items to output this cycle E1121 19:55:26.412639 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:55:26.412675 3798194 memory.go:191] Add success. I1121 19:55:26.417300 3798194 cpu.go:282] Add success. I1121 19:55:26.441736 3798194 net.go:642] Add success. I1121 19:55:26.443155 3798194 net.go:778] primary dev: ETH0 I1121 19:55:26.443169 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:55:26.443182 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:55:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:55:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:55:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:55:36.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:55:36.411992 3798194 memory.go:184] no items to output this cycle I1121 19:55:36.412145 3798194 cpu.go:275] no items to output this cycle E1121 19:55:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:55:46.411975 3798194 memory.go:184] no items to output this cycle I1121 19:55:46.412130 3798194 cpu.go:275] no items to output this cycle E1121 19:55:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:55:56.411952 3798194 memory.go:191] Add success. W1121 19:55:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:55:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:55:56.412000 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:55:56.412344 3798194 cpu.go:282] Add success. I1121 19:55:56.441567 3798194 event_worker.go:152] Polling the log file for events... I1121 19:55:56.444681 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:55:56.445231 3798194 net.go:642] Add success. I1121 19:55:56.449343 3798194 net.go:778] primary dev: ETH0 I1121 19:55:56.449362 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:55:56.449375 3798194 net.go:659] Add eni metrics success, count: 7 E1121 19:55:57.450583 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 19:55:57.450596 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:55:57.450762 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:55:57.450818 3798194 disk_worker.go:723] disk space is not compliant W1121 19:55:57.450822 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:55:57.459107 3798194 disk_worker.go:509] system disk:vda1 I1121 19:55:57.459162 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 19:55:58.448088 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 19:55:58.448102 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:55:59.338136 3798194 disk_info.go:125] begin check local disk info of client I1121 19:55:59.342028 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:55:59.342036 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000460100 0xc000460140] E1121 19:55:59.449008 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 19:55:59.450083 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:55:59.450142 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:55:59.450167 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:55:59.471683 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:56:06.413044 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:56:06.413069 3798194 memory.go:184] no items to output this cycle I1121 19:56:06.413620 3798194 cpu.go:275] no items to output this cycle E1121 19:56:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:56:16.411980 3798194 memory.go:184] no items to output this cycle I1121 19:56:16.412146 3798194 cpu.go:275] no items to output this cycle E1121 19:56:26.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:56:26.412899 3798194 memory.go:191] Add success. I1121 19:56:26.417291 3798194 cpu.go:282] Add success. I1121 19:56:26.442246 3798194 net.go:642] Add success. I1121 19:56:26.443722 3798194 net.go:778] primary dev: ETH0 I1121 19:56:26.443737 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:56:26.443751 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:56:29.450008 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:56:29.450085 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:56:29.450125 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:56:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:56:36.411984 3798194 memory.go:184] no items to output this cycle I1121 19:56:36.412139 3798194 cpu.go:275] no items to output this cycle E1121 19:56:46.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:56:46.413263 3798194 memory.go:184] no items to output this cycle I1121 19:56:46.413416 3798194 cpu.go:275] no items to output this cycle E1121 19:56:56.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:56:56.413278 3798194 memory.go:191] Add success. W1121 19:56:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:56:56.413323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:56:56.413326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:56:56.417826 3798194 cpu.go:282] Add success. I1121 19:56:56.441359 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:56:56.441905 3798194 net.go:642] Add success. I1121 19:56:56.443484 3798194 net.go:778] primary dev: ETH0 I1121 19:56:56.443499 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:56:56.443512 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:56:57.450773 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:56:57.450936 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:56:57.450990 3798194 disk_worker.go:723] disk space is not compliant W1121 19:56:57.450993 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:56:57.459815 3798194 disk_worker.go:509] system disk:vda1 I1121 19:56:57.459858 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:56:58.446383 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:56:59.342143 3798194 disk_info.go:125] begin check local disk info of client I1121 19:56:59.346060 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:56:59.346069 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000371240 0xc000371280] I1121 19:56:59.449664 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:56:59.449737 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:56:59.449761 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:56:59.471364 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:57:06.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:57:06.413256 3798194 memory.go:184] no items to output this cycle I1121 19:57:06.413395 3798194 cpu.go:275] no items to output this cycle E1121 19:57:16.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:57:16.411990 3798194 memory.go:184] no items to output this cycle I1121 19:57:16.412147 3798194 cpu.go:275] no items to output this cycle I1121 19:57:17.548775 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 19:57:17.548783 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 19:57:26.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:57:26.412273 3798194 memory.go:191] Add success. I1121 19:57:26.416733 3798194 cpu.go:282] Add success. I1121 19:57:26.441601 3798194 net.go:642] Add success. I1121 19:57:26.443675 3798194 net.go:778] primary dev: ETH0 I1121 19:57:26.443696 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:57:26.443709 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:57:29.450385 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:57:29.450458 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:57:29.450486 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:57:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:57:36.413262 3798194 memory.go:184] no items to output this cycle I1121 19:57:36.413401 3798194 cpu.go:275] no items to output this cycle E1121 19:57:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:57:46.411973 3798194 memory.go:184] no items to output this cycle I1121 19:57:46.412132 3798194 cpu.go:275] no items to output this cycle E1121 19:57:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:57:56.411950 3798194 memory.go:191] Add success. W1121 19:57:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:57:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:57:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:57:56.412348 3798194 cpu.go:282] Add success. I1121 19:57:56.441833 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:57:56.442436 3798194 net.go:642] Add success. I1121 19:57:56.446563 3798194 net.go:778] primary dev: ETH0 I1121 19:57:56.446589 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:57:56.446603 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:57:57.450665 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:57:57.450824 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:57:57.450875 3798194 disk_worker.go:723] disk space is not compliant W1121 19:57:57.450878 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:57:57.459215 3798194 disk_worker.go:509] system disk:vda1 I1121 19:57:57.459260 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:57:58.446384 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:57:59.349146 3798194 disk_info.go:125] begin check local disk info of client I1121 19:57:59.352680 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:57:59.352689 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003eb640 0xc0003eb680] I1121 19:57:59.450186 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:57:59.450262 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:57:59.450285 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:57:59.470827 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:58:06.413179 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:58:06.413201 3798194 memory.go:184] no items to output this cycle I1121 19:58:06.413345 3798194 cpu.go:275] no items to output this cycle E1121 19:58:16.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:58:16.412257 3798194 memory.go:184] no items to output this cycle I1121 19:58:16.412411 3798194 cpu.go:275] no items to output this cycle E1121 19:58:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:58:26.411958 3798194 memory.go:191] Add success. I1121 19:58:26.412314 3798194 cpu.go:282] Add success. I1121 19:58:26.441952 3798194 net.go:642] Add success. I1121 19:58:26.446460 3798194 net.go:778] primary dev: ETH0 I1121 19:58:26.446475 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:58:26.446490 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:58:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:58:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:58:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:58:36.412359 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:58:36.412382 3798194 memory.go:184] no items to output this cycle I1121 19:58:36.412525 3798194 cpu.go:275] no items to output this cycle E1121 19:58:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:58:46.411976 3798194 memory.go:184] no items to output this cycle I1121 19:58:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 19:58:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:58:56.412274 3798194 memory.go:191] Add success. W1121 19:58:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:58:56.412320 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:58:56.412323 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:58:56.416803 3798194 cpu.go:282] Add success. I1121 19:58:56.441386 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:58:56.441873 3798194 net.go:642] Add success. I1121 19:58:56.443477 3798194 net.go:778] primary dev: ETH0 I1121 19:58:56.443492 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:58:56.443506 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:58:57.450931 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:58:57.451168 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:58:57.451180 3798194 disk_worker.go:723] disk space is not compliant W1121 19:58:57.451182 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:58:57.459936 3798194 disk_worker.go:509] system disk:vda1 I1121 19:58:57.459991 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:58:58.446015 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:58:59.355340 3798194 disk_info.go:125] begin check local disk info of client I1121 19:58:59.359197 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:58:59.359205 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0e40 0xc0002e0e80] I1121 19:58:59.449674 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:58:59.449752 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:58:59.449779 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:58:59.471339 3798194 disk_local_worker.go:436] Get disk info: [] E1121 19:59:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:59:06.411989 3798194 memory.go:184] no items to output this cycle I1121 19:59:06.412149 3798194 cpu.go:275] no items to output this cycle E1121 19:59:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:59:16.411976 3798194 memory.go:184] no items to output this cycle I1121 19:59:16.412137 3798194 cpu.go:275] no items to output this cycle E1121 19:59:26.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:59:26.413276 3798194 memory.go:191] Add success. I1121 19:59:26.417619 3798194 cpu.go:282] Add success. I1121 19:59:26.442203 3798194 net.go:642] Add success. I1121 19:59:26.443585 3798194 net.go:778] primary dev: ETH0 I1121 19:59:26.443600 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:59:26.443612 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:59:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:59:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:59:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 19:59:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:59:36.411982 3798194 memory.go:184] no items to output this cycle I1121 19:59:36.412136 3798194 cpu.go:275] no items to output this cycle E1121 19:59:46.411933 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:59:46.411955 3798194 memory.go:184] no items to output this cycle I1121 19:59:46.412126 3798194 cpu.go:275] no items to output this cycle E1121 19:59:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 19:59:56.411951 3798194 memory.go:191] Add success. W1121 19:59:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 19:59:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 19:59:56.411993 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 19:59:56.412329 3798194 cpu.go:282] Add success. I1121 19:59:56.441790 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 19:59:56.442292 3798194 net.go:642] Add success. I1121 19:59:56.446819 3798194 net.go:778] primary dev: ETH0 I1121 19:59:56.446841 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 19:59:56.446855 3798194 net.go:659] Add eni metrics success, count: 7 I1121 19:59:57.451255 3798194 custom_config.go:106] query custom config with name: gpu W1121 19:59:57.451434 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 19:59:57.451488 3798194 disk_worker.go:723] disk space is not compliant W1121 19:59:57.451491 3798194 disk_worker.go:743] disk inode is not compliant I1121 19:59:57.461085 3798194 disk_worker.go:509] system disk:vda1 I1121 19:59:57.461146 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 19:59:58.446373 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 19:59:59.362704 3798194 disk_info.go:125] begin check local disk info of client I1121 19:59:59.366552 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 19:59:59.366560 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00045f800 0xc00045f840] I1121 19:59:59.449866 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 19:59:59.449928 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 19:59:59.449952 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 19:59:59.471496 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:00:06.412762 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:00:06.412784 3798194 memory.go:184] no items to output this cycle I1121 20:00:06.412945 3798194 cpu.go:275] no items to output this cycle E1121 20:00:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:00:16.411986 3798194 memory.go:184] no items to output this cycle I1121 20:00:16.412152 3798194 cpu.go:275] no items to output this cycle I1121 20:00:17.548953 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:00:17.548962 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:00:26.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:00:26.412279 3798194 memory.go:191] Add success. I1121 20:00:26.416243 3798194 cpu.go:282] Add success. I1121 20:00:26.441759 3798194 net.go:642] Add success. I1121 20:00:26.443330 3798194 net.go:778] primary dev: ETH0 I1121 20:00:26.443346 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:00:26.443360 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:00:29.449865 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:00:29.449949 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:00:29.449973 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:00:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:00:36.411978 3798194 memory.go:184] no items to output this cycle I1121 20:00:36.412142 3798194 cpu.go:275] no items to output this cycle E1121 20:00:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:00:46.411983 3798194 memory.go:184] no items to output this cycle I1121 20:00:46.412151 3798194 cpu.go:275] no items to output this cycle E1121 20:00:56.412881 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:00:56.412915 3798194 memory.go:191] Add success. W1121 20:00:56.412948 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:00:56.412959 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:00:56.412963 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:00:56.417083 3798194 cpu.go:282] Add success. I1121 20:00:56.444502 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:00:56.444999 3798194 net.go:642] Add success. I1121 20:00:56.446567 3798194 net.go:778] primary dev: ETH0 I1121 20:00:56.446582 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:00:56.446593 3798194 net.go:659] Add eni metrics success, count: 7 E1121 20:00:57.450661 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 20:00:57.450676 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:00:57.450895 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:00:57.450905 3798194 disk_worker.go:723] disk space is not compliant W1121 20:00:57.450908 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:00:57.459316 3798194 disk_worker.go:509] system disk:vda1 I1121 20:00:57.459358 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 20:00:58.447975 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 20:00:58.447989 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:00:59.370612 3798194 disk_info.go:125] begin check local disk info of client I1121 20:00:59.374426 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:00:59.374435 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a0b80 0xc0003a0bc0] E1121 20:00:59.448429 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 20:00:59.450558 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:00:59.450608 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:00:59.450626 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:00:59.471127 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:01:06.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:01:06.412880 3798194 memory.go:184] no items to output this cycle I1121 20:01:06.413022 3798194 cpu.go:275] no items to output this cycle E1121 20:01:16.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:01:16.412875 3798194 memory.go:184] no items to output this cycle I1121 20:01:16.413019 3798194 cpu.go:275] no items to output this cycle E1121 20:01:26.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:01:26.413285 3798194 memory.go:191] Add success. I1121 20:01:26.417565 3798194 cpu.go:282] Add success. I1121 20:01:26.442245 3798194 net.go:642] Add success. I1121 20:01:26.443676 3798194 net.go:778] primary dev: ETH0 I1121 20:01:26.443690 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:01:26.443702 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:01:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:01:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:01:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:01:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:01:36.413256 3798194 memory.go:184] no items to output this cycle I1121 20:01:36.413402 3798194 cpu.go:275] no items to output this cycle E1121 20:01:46.411980 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:01:46.412001 3798194 memory.go:184] no items to output this cycle I1121 20:01:46.412160 3798194 cpu.go:275] no items to output this cycle E1121 20:01:56.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:01:56.411958 3798194 memory.go:191] Add success. W1121 20:01:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:01:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:01:56.412000 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:01:56.412359 3798194 cpu.go:282] Add success. I1121 20:01:56.440923 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:01:56.441524 3798194 net.go:642] Add success. I1121 20:01:56.445375 3798194 net.go:778] primary dev: ETH0 I1121 20:01:56.445390 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:01:56.445402 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:01:57.445582 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:01:57.445791 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:01:57.445855 3798194 disk_worker.go:723] disk space is not compliant W1121 20:01:57.445858 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:01:57.450829 3798194 disk_worker.go:509] system disk:vda1 I1121 20:01:57.450875 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:01:58.446466 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:01:59.374529 3798194 disk_info.go:125] begin check local disk info of client I1121 20:01:59.378395 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:01:59.378405 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002f3340 0xc0002f3380] I1121 20:01:59.449672 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:01:59.449747 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:01:59.449772 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:01:59.471276 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:02:06.411467 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:02:06.411492 3798194 memory.go:184] no items to output this cycle I1121 20:02:06.411639 3798194 cpu.go:275] no items to output this cycle E1121 20:02:16.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:02:16.413259 3798194 memory.go:184] no items to output this cycle I1121 20:02:16.413408 3798194 cpu.go:275] no items to output this cycle E1121 20:02:26.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:02:26.412887 3798194 memory.go:191] Add success. I1121 20:02:26.417385 3798194 cpu.go:282] Add success. I1121 20:02:26.441269 3798194 net.go:642] Add success. I1121 20:02:26.442784 3798194 net.go:778] primary dev: ETH0 I1121 20:02:26.442799 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:02:26.442813 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:02:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:02:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:02:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:02:36.412574 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:02:36.412595 3798194 memory.go:184] no items to output this cycle I1121 20:02:36.412735 3798194 cpu.go:275] no items to output this cycle E1121 20:02:46.412998 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:02:46.413025 3798194 memory.go:184] no items to output this cycle I1121 20:02:46.413189 3798194 cpu.go:275] no items to output this cycle E1121 20:02:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:02:56.411951 3798194 memory.go:191] Add success. W1121 20:02:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:02:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:02:56.411993 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:02:56.412363 3798194 cpu.go:282] Add success. I1121 20:02:56.441138 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:02:56.441624 3798194 net.go:642] Add success. I1121 20:02:56.445632 3798194 net.go:778] primary dev: ETH0 I1121 20:02:56.445647 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:02:56.445659 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:02:57.450278 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:02:57.450486 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:02:57.450497 3798194 disk_worker.go:723] disk space is not compliant W1121 20:02:57.450500 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:02:57.458666 3798194 disk_worker.go:509] system disk:vda1 I1121 20:02:57.458724 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:02:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:02:59.378510 3798194 disk_info.go:125] begin check local disk info of client I1121 20:02:59.382461 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:02:59.382472 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036ccc0 0xc00036cd00] I1121 20:02:59.449969 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:02:59.450053 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:02:59.450078 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:02:59.471637 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:03:06.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:03:06.413267 3798194 memory.go:184] no items to output this cycle I1121 20:03:06.413406 3798194 cpu.go:275] no items to output this cycle E1121 20:03:16.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:03:16.413257 3798194 memory.go:184] no items to output this cycle I1121 20:03:16.413403 3798194 cpu.go:275] no items to output this cycle I1121 20:03:17.550053 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:03:17.550062 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:03:26.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:03:26.413272 3798194 memory.go:191] Add success. I1121 20:03:26.417629 3798194 cpu.go:282] Add success. I1121 20:03:26.441414 3798194 net.go:642] Add success. I1121 20:03:26.442929 3798194 net.go:778] primary dev: ETH0 I1121 20:03:26.442943 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:03:26.442966 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:03:29.450205 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:03:29.450283 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:03:29.450310 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:03:36.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:03:36.411972 3798194 memory.go:184] no items to output this cycle I1121 20:03:36.412124 3798194 cpu.go:275] no items to output this cycle E1121 20:03:46.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:03:46.413260 3798194 memory.go:184] no items to output this cycle I1121 20:03:46.413408 3798194 cpu.go:275] no items to output this cycle E1121 20:03:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:03:56.411949 3798194 memory.go:191] Add success. W1121 20:03:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:03:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:03:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:03:56.412360 3798194 cpu.go:282] Add success. I1121 20:03:56.441961 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:03:56.442700 3798194 net.go:642] Add success. I1121 20:03:56.446496 3798194 net.go:778] primary dev: ETH0 I1121 20:03:56.446520 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:03:56.446534 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:03:57.450666 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:03:57.450825 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:03:57.450883 3798194 disk_worker.go:723] disk space is not compliant W1121 20:03:57.450886 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:03:57.459155 3798194 disk_worker.go:509] system disk:vda1 I1121 20:03:57.459196 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:03:58.449171 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:03:59.386469 3798194 disk_info.go:125] begin check local disk info of client I1121 20:03:59.390391 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:03:59.390401 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e380 0xc00030e3c0] I1121 20:03:59.449668 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:03:59.449726 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:03:59.449751 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:03:59.471244 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:04:06.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:04:06.411986 3798194 memory.go:184] no items to output this cycle I1121 20:04:06.412144 3798194 cpu.go:275] no items to output this cycle E1121 20:04:16.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:04:16.411971 3798194 memory.go:184] no items to output this cycle I1121 20:04:16.412149 3798194 cpu.go:275] no items to output this cycle E1121 20:04:26.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:04:26.413262 3798194 memory.go:191] Add success. I1121 20:04:26.417468 3798194 cpu.go:282] Add success. I1121 20:04:26.442146 3798194 net.go:642] Add success. I1121 20:04:26.443987 3798194 net.go:778] primary dev: ETH0 I1121 20:04:26.444002 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:04:26.444016 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:04:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:04:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:04:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:04:36.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:04:36.411942 3798194 memory.go:184] no items to output this cycle I1121 20:04:36.412083 3798194 cpu.go:275] no items to output this cycle E1121 20:04:46.411098 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:04:46.411139 3798194 memory.go:184] no items to output this cycle I1121 20:04:46.411295 3798194 cpu.go:275] no items to output this cycle E1121 20:04:56.412788 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:04:56.412824 3798194 memory.go:191] Add success. W1121 20:04:56.412856 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:04:56.412867 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:04:56.412871 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:04:56.417282 3798194 cpu.go:282] Add success. I1121 20:04:56.441826 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:04:56.442430 3798194 net.go:642] Add success. I1121 20:04:56.443940 3798194 net.go:778] primary dev: ETH0 I1121 20:04:56.443955 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:04:56.443968 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:04:57.450638 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:04:57.450797 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:04:57.450852 3798194 disk_worker.go:723] disk space is not compliant W1121 20:04:57.450855 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:04:57.459186 3798194 disk_worker.go:509] system disk:vda1 I1121 20:04:57.459228 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:04:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:04:59.395138 3798194 disk_info.go:125] begin check local disk info of client I1121 20:04:59.399025 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:04:59.399034 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c9e80 0xc0003c9ec0] I1121 20:04:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:04:59.450462 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:04:59.450484 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:04:59.470986 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:05:06.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:05:06.412881 3798194 memory.go:184] no items to output this cycle I1121 20:05:06.413038 3798194 cpu.go:275] no items to output this cycle E1121 20:05:16.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:05:16.413254 3798194 memory.go:184] no items to output this cycle I1121 20:05:16.413400 3798194 cpu.go:275] no items to output this cycle E1121 20:05:26.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:05:26.412905 3798194 memory.go:191] Add success. I1121 20:05:26.417183 3798194 cpu.go:282] Add success. I1121 20:05:26.441757 3798194 net.go:642] Add success. I1121 20:05:26.443231 3798194 net.go:778] primary dev: ETH0 I1121 20:05:26.443245 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:05:26.443259 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:05:29.450003 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:05:29.450086 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:05:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:05:36.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:05:36.411990 3798194 memory.go:184] no items to output this cycle I1121 20:05:36.412144 3798194 cpu.go:275] no items to output this cycle E1121 20:05:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:05:46.411981 3798194 memory.go:184] no items to output this cycle I1121 20:05:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 20:05:56.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:05:56.411946 3798194 memory.go:191] Add success. W1121 20:05:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:05:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:05:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:05:56.412349 3798194 cpu.go:282] Add success. I1121 20:05:56.441693 3798194 event_worker.go:152] Polling the log file for events... I1121 20:05:56.445289 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:05:56.445758 3798194 net.go:642] Add success. I1121 20:05:56.456935 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795260956, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 20:05:56.457066 3798194 net.go:778] primary dev: ETH0 I1121 20:05:56.457082 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:05:56.457093 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:05:56.464924 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 20:05:57.445580 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 20:05:57.445606 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:05:57.445802 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:05:57.445878 3798194 disk_worker.go:723] disk space is not compliant W1121 20:05:57.445881 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:05:57.450847 3798194 disk_worker.go:509] system disk:vda1 I1121 20:05:57.450887 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 20:05:58.448108 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 20:05:58.448138 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:05:59.399143 3798194 disk_info.go:125] begin check local disk info of client I1121 20:05:59.403006 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:05:59.403015 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000331680 0xc0003316c0] E1121 20:05:59.448634 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 20:05:59.449709 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:05:59.449757 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:05:59.449782 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:05:59.471329 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:06:06.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:06:06.412267 3798194 memory.go:184] no items to output this cycle I1121 20:06:06.412414 3798194 cpu.go:275] no items to output this cycle E1121 20:06:16.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:06:16.412883 3798194 memory.go:184] no items to output this cycle I1121 20:06:16.413062 3798194 cpu.go:275] no items to output this cycle I1121 20:06:17.550228 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:06:17.550235 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:06:26.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:06:26.413277 3798194 memory.go:191] Add success. I1121 20:06:26.417503 3798194 cpu.go:282] Add success. I1121 20:06:26.442044 3798194 net.go:642] Add success. I1121 20:06:26.443712 3798194 net.go:778] primary dev: ETH0 I1121 20:06:26.443727 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:06:26.443740 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:06:29.450543 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:06:29.450623 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:06:29.450650 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:06:36.413377 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:06:36.413399 3798194 memory.go:184] no items to output this cycle I1121 20:06:36.413541 3798194 cpu.go:275] no items to output this cycle E1121 20:06:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:06:46.411982 3798194 memory.go:184] no items to output this cycle I1121 20:06:46.412344 3798194 cpu.go:275] no items to output this cycle E1121 20:06:56.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:06:56.412892 3798194 memory.go:191] Add success. W1121 20:06:56.412925 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:06:56.412936 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:06:56.412940 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:06:56.417158 3798194 cpu.go:282] Add success. I1121 20:06:56.441558 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:06:56.442026 3798194 net.go:642] Add success. I1121 20:06:56.443545 3798194 net.go:778] primary dev: ETH0 I1121 20:06:56.443561 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:06:56.443573 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:06:57.450755 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:06:57.450979 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:06:57.450989 3798194 disk_worker.go:723] disk space is not compliant W1121 20:06:57.450992 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:06:57.460093 3798194 disk_worker.go:509] system disk:vda1 I1121 20:06:57.460152 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:06:58.445996 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:06:59.407148 3798194 disk_info.go:125] begin check local disk info of client I1121 20:06:59.410966 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:06:59.410975 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000301080 0xc0003010c0] I1121 20:06:59.450185 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:06:59.450237 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:06:59.450263 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:06:59.470729 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:07:06.412246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:07:06.412268 3798194 memory.go:184] no items to output this cycle I1121 20:07:06.412415 3798194 cpu.go:275] no items to output this cycle E1121 20:07:16.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:07:16.413271 3798194 memory.go:184] no items to output this cycle I1121 20:07:16.413428 3798194 cpu.go:275] no items to output this cycle E1121 20:07:26.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:07:26.412892 3798194 memory.go:191] Add success. I1121 20:07:26.417375 3798194 cpu.go:282] Add success. I1121 20:07:26.441908 3798194 net.go:642] Add success. I1121 20:07:26.443620 3798194 net.go:778] primary dev: ETH0 I1121 20:07:26.443634 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:07:26.443647 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:07:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:07:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:07:29.450514 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:07:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:07:36.413265 3798194 memory.go:184] no items to output this cycle I1121 20:07:36.413408 3798194 cpu.go:275] no items to output this cycle E1121 20:07:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:07:46.411981 3798194 memory.go:184] no items to output this cycle I1121 20:07:46.412138 3798194 cpu.go:275] no items to output this cycle E1121 20:07:56.411929 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:07:56.411964 3798194 memory.go:191] Add success. W1121 20:07:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:07:56.412011 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:07:56.412014 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:07:56.412366 3798194 cpu.go:282] Add success. I1121 20:07:56.441856 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:07:56.442472 3798194 net.go:642] Add success. I1121 20:07:56.446584 3798194 net.go:778] primary dev: ETH0 I1121 20:07:56.446600 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:07:56.446615 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:07:57.450980 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:07:57.451181 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:07:57.451253 3798194 disk_worker.go:723] disk space is not compliant W1121 20:07:57.451258 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:07:57.460126 3798194 disk_worker.go:509] system disk:vda1 I1121 20:07:57.460170 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:07:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:07:59.412761 3798194 disk_info.go:125] begin check local disk info of client I1121 20:07:59.416593 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:07:59.416603 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004c2d00 0xc0004c2d40] I1121 20:07:59.449848 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:07:59.449906 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:07:59.449929 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:07:59.471527 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:08:06.412772 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:08:06.412794 3798194 memory.go:184] no items to output this cycle I1121 20:08:06.412946 3798194 cpu.go:275] no items to output this cycle E1121 20:08:16.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:08:16.413267 3798194 memory.go:184] no items to output this cycle I1121 20:08:16.413411 3798194 cpu.go:275] no items to output this cycle E1121 20:08:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:08:26.411947 3798194 memory.go:191] Add success. I1121 20:08:26.412311 3798194 cpu.go:282] Add success. I1121 20:08:26.441914 3798194 net.go:642] Add success. I1121 20:08:26.445991 3798194 net.go:778] primary dev: ETH0 I1121 20:08:26.446008 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:08:26.446022 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:08:29.450378 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:08:29.450455 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:08:29.450480 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:08:36.412953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:08:36.412974 3798194 memory.go:184] no items to output this cycle I1121 20:08:36.413140 3798194 cpu.go:275] no items to output this cycle E1121 20:08:46.413304 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:08:46.413325 3798194 memory.go:184] no items to output this cycle I1121 20:08:46.413471 3798194 cpu.go:275] no items to output this cycle E1121 20:08:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:08:56.411947 3798194 memory.go:191] Add success. W1121 20:08:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:08:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:08:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:08:56.412355 3798194 cpu.go:282] Add success. I1121 20:08:56.441803 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:08:56.442307 3798194 net.go:642] Add success. I1121 20:08:56.446708 3798194 net.go:778] primary dev: ETH0 I1121 20:08:56.446724 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:08:56.446737 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:08:57.450981 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:08:57.451166 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:08:57.451224 3798194 disk_worker.go:723] disk space is not compliant W1121 20:08:57.451227 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:08:57.460065 3798194 disk_worker.go:509] system disk:vda1 I1121 20:08:57.460108 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:08:58.445697 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:08:59.416863 3798194 disk_info.go:125] begin check local disk info of client I1121 20:08:59.420755 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:08:59.420764 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b8040 0xc0003b8080] I1121 20:08:59.449982 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:08:59.450031 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:08:59.450053 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:08:59.471585 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:09:06.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:09:06.413264 3798194 memory.go:184] no items to output this cycle I1121 20:09:06.413402 3798194 cpu.go:275] no items to output this cycle E1121 20:09:16.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:09:16.413257 3798194 memory.go:184] no items to output this cycle I1121 20:09:16.413401 3798194 cpu.go:275] no items to output this cycle I1121 20:09:17.550402 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:09:17.550411 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:09:26.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:09:26.413265 3798194 memory.go:191] Add success. I1121 20:09:26.417639 3798194 cpu.go:282] Add success. I1121 20:09:26.441270 3798194 net.go:642] Add success. I1121 20:09:26.442791 3798194 net.go:778] primary dev: ETH0 I1121 20:09:26.442806 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:09:26.442819 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:09:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:09:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:09:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:09:36.411971 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:09:36.411995 3798194 memory.go:184] no items to output this cycle I1121 20:09:36.412150 3798194 cpu.go:275] no items to output this cycle E1121 20:09:46.412986 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:09:46.413009 3798194 memory.go:184] no items to output this cycle I1121 20:09:46.413176 3798194 cpu.go:275] no items to output this cycle E1121 20:09:56.412868 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:09:56.412904 3798194 memory.go:191] Add success. W1121 20:09:56.412936 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:09:56.412948 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:09:56.412951 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:09:56.417866 3798194 cpu.go:282] Add success. I1121 20:09:56.441366 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:09:56.441940 3798194 net.go:642] Add success. I1121 20:09:56.443512 3798194 net.go:778] primary dev: ETH0 I1121 20:09:56.443526 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:09:56.443539 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:09:57.450976 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:09:57.451205 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:09:57.451216 3798194 disk_worker.go:723] disk space is not compliant W1121 20:09:57.451219 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:09:57.460044 3798194 disk_worker.go:509] system disk:vda1 I1121 20:09:57.460089 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:09:58.446684 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:09:59.420825 3798194 disk_info.go:125] begin check local disk info of client I1121 20:09:59.424676 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:09:59.424685 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000460dc0 0xc000460e00] I1121 20:09:59.449887 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:09:59.449939 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:09:59.449961 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:09:59.471504 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:10:06.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:10:06.411975 3798194 memory.go:184] no items to output this cycle I1121 20:10:06.412131 3798194 cpu.go:275] no items to output this cycle E1121 20:10:16.412734 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:10:16.412757 3798194 memory.go:184] no items to output this cycle I1121 20:10:16.412903 3798194 cpu.go:275] no items to output this cycle E1121 20:10:26.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:10:26.412891 3798194 memory.go:191] Add success. I1121 20:10:26.417880 3798194 cpu.go:282] Add success. I1121 20:10:26.441424 3798194 net.go:642] Add success. I1121 20:10:26.442994 3798194 net.go:778] primary dev: ETH0 I1121 20:10:26.443010 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:10:26.443022 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:10:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:10:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:10:29.450496 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:10:36.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:10:36.411990 3798194 memory.go:184] no items to output this cycle I1121 20:10:36.412153 3798194 cpu.go:275] no items to output this cycle E1121 20:10:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:10:46.411974 3798194 memory.go:184] no items to output this cycle I1121 20:10:46.412138 3798194 cpu.go:275] no items to output this cycle E1121 20:10:56.413094 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:10:56.413140 3798194 memory.go:191] Add success. W1121 20:10:56.413168 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:10:56.413179 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:10:56.413182 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:10:56.417590 3798194 cpu.go:282] Add success. I1121 20:10:56.442417 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:10:56.442898 3798194 net.go:642] Add success. I1121 20:10:56.444852 3798194 net.go:778] primary dev: ETH0 I1121 20:10:56.444867 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:10:56.444882 3798194 net.go:659] Add eni metrics success, count: 7 E1121 20:10:57.449336 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 20:10:57.449351 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:10:57.449761 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:10:57.449772 3798194 disk_worker.go:723] disk space is not compliant W1121 20:10:57.449775 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:10:57.452186 3798194 disk_worker.go:509] system disk:vda1 I1121 20:10:57.452224 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 20:10:58.448528 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 20:10:58.448543 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:10:59.425279 3798194 disk_info.go:125] begin check local disk info of client I1121 20:10:59.429148 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:10:59.429157 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004bba00 0xc0004bba40] E1121 20:10:59.448746 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 20:10:59.449819 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:10:59.449864 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:10:59.449880 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:10:59.471398 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:11:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:11:06.411987 3798194 memory.go:184] no items to output this cycle I1121 20:11:06.412149 3798194 cpu.go:275] no items to output this cycle E1121 20:11:16.413187 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:11:16.413212 3798194 memory.go:184] no items to output this cycle I1121 20:11:16.413354 3798194 cpu.go:275] no items to output this cycle E1121 20:11:26.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:11:26.413276 3798194 memory.go:191] Add success. I1121 20:11:26.417585 3798194 cpu.go:282] Add success. I1121 20:11:26.442185 3798194 net.go:642] Add success. I1121 20:11:26.443630 3798194 net.go:778] primary dev: ETH0 I1121 20:11:26.443646 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:11:26.443659 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:11:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:11:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:11:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:11:36.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:11:36.411987 3798194 memory.go:184] no items to output this cycle I1121 20:11:36.412142 3798194 cpu.go:275] no items to output this cycle E1121 20:11:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:11:46.412257 3798194 memory.go:184] no items to output this cycle I1121 20:11:46.412400 3798194 cpu.go:275] no items to output this cycle E1121 20:11:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:11:56.411956 3798194 memory.go:191] Add success. W1121 20:11:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:11:56.412001 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:11:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:11:56.412364 3798194 cpu.go:282] Add success. I1121 20:11:56.441825 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:11:56.442333 3798194 net.go:642] Add success. I1121 20:11:56.446854 3798194 net.go:778] primary dev: ETH0 I1121 20:11:56.446869 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:11:56.446881 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:11:57.450583 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:11:57.450786 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:11:57.450798 3798194 disk_worker.go:723] disk space is not compliant W1121 20:11:57.450802 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:11:57.459146 3798194 disk_worker.go:509] system disk:vda1 I1121 20:11:57.459189 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:11:58.445688 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:11:59.429915 3798194 disk_info.go:125] begin check local disk info of client I1121 20:11:59.433605 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:11:59.433617 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e6cc0 0xc0003e6d00] I1121 20:11:59.449769 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:11:59.449819 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:11:59.449839 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:11:59.471380 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:12:06.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:12:06.413251 3798194 memory.go:184] no items to output this cycle I1121 20:12:06.413393 3798194 cpu.go:275] no items to output this cycle E1121 20:12:16.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:12:16.412888 3798194 memory.go:184] no items to output this cycle I1121 20:12:16.413035 3798194 cpu.go:275] no items to output this cycle I1121 20:12:17.550586 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:12:17.550594 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:12:26.411391 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:12:26.411428 3798194 memory.go:191] Add success. I1121 20:12:26.411791 3798194 cpu.go:282] Add success. I1121 20:12:26.441447 3798194 net.go:642] Add success. I1121 20:12:26.445515 3798194 net.go:778] primary dev: ETH0 I1121 20:12:26.445531 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:12:26.445544 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:12:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:12:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:12:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:12:36.412452 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:12:36.412473 3798194 memory.go:184] no items to output this cycle I1121 20:12:36.412615 3798194 cpu.go:275] no items to output this cycle E1121 20:12:46.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:12:46.413267 3798194 memory.go:184] no items to output this cycle I1121 20:12:46.413417 3798194 cpu.go:275] no items to output this cycle E1121 20:12:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:12:56.411953 3798194 memory.go:191] Add success. W1121 20:12:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:12:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:12:56.412000 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:12:56.412343 3798194 cpu.go:282] Add success. I1121 20:12:56.441830 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:12:56.442331 3798194 net.go:642] Add success. I1121 20:12:56.446438 3798194 net.go:778] primary dev: ETH0 I1121 20:12:56.446452 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:12:56.446465 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:12:57.450599 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:12:57.450813 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:12:57.450824 3798194 disk_worker.go:723] disk space is not compliant W1121 20:12:57.450827 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:12:57.459162 3798194 disk_worker.go:509] system disk:vda1 I1121 20:12:57.459202 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:12:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:12:59.433945 3798194 disk_info.go:125] begin check local disk info of client I1121 20:12:59.437710 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:12:59.437719 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000326b40 0xc000326b80] I1121 20:12:59.449853 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:12:59.449903 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:12:59.449925 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:12:59.471446 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:13:06.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:13:06.411978 3798194 memory.go:184] no items to output this cycle I1121 20:13:06.412143 3798194 cpu.go:275] no items to output this cycle E1121 20:13:16.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:13:16.413248 3798194 memory.go:184] no items to output this cycle I1121 20:13:16.413394 3798194 cpu.go:275] no items to output this cycle E1121 20:13:26.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:13:26.413273 3798194 memory.go:191] Add success. I1121 20:13:26.417562 3798194 cpu.go:282] Add success. I1121 20:13:26.442094 3798194 net.go:642] Add success. I1121 20:13:26.443605 3798194 net.go:778] primary dev: ETH0 I1121 20:13:26.443621 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:13:26.443635 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:13:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:13:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:13:29.450496 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:13:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:13:36.411976 3798194 memory.go:184] no items to output this cycle I1121 20:13:36.412136 3798194 cpu.go:275] no items to output this cycle E1121 20:13:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:13:46.411981 3798194 memory.go:184] no items to output this cycle I1121 20:13:46.412142 3798194 cpu.go:275] no items to output this cycle E1121 20:13:56.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:13:56.412269 3798194 memory.go:191] Add success. W1121 20:13:56.412304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:13:56.412316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:13:56.412319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:13:56.416988 3798194 cpu.go:282] Add success. I1121 20:13:56.440977 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:13:56.441496 3798194 net.go:642] Add success. I1121 20:13:56.443063 3798194 net.go:778] primary dev: ETH0 I1121 20:13:56.443077 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:13:56.443092 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:13:57.451418 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:13:57.451582 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:13:57.451636 3798194 disk_worker.go:723] disk space is not compliant W1121 20:13:57.451640 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:13:57.459953 3798194 disk_worker.go:509] system disk:vda1 I1121 20:13:57.459994 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:13:58.446368 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:13:59.438374 3798194 disk_info.go:125] begin check local disk info of client I1121 20:13:59.443749 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:13:59.443761 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e380 0xc00030e3c0] I1121 20:13:59.449912 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:13:59.449985 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:13:59.450016 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:13:59.474155 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:14:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:14:06.411982 3798194 memory.go:184] no items to output this cycle I1121 20:14:06.412133 3798194 cpu.go:275] no items to output this cycle E1121 20:14:16.413029 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:14:16.413054 3798194 memory.go:184] no items to output this cycle I1121 20:14:16.413220 3798194 cpu.go:275] no items to output this cycle E1121 20:14:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:14:26.411956 3798194 memory.go:191] Add success. I1121 20:14:26.412314 3798194 cpu.go:282] Add success. I1121 20:14:26.441757 3798194 net.go:642] Add success. I1121 20:14:26.446178 3798194 net.go:778] primary dev: ETH0 I1121 20:14:26.446195 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:14:26.446209 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:14:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:14:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:14:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:14:36.412767 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:14:36.412789 3798194 memory.go:184] no items to output this cycle I1121 20:14:36.412939 3798194 cpu.go:275] no items to output this cycle E1121 20:14:46.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:14:46.413262 3798194 memory.go:184] no items to output this cycle I1121 20:14:46.413404 3798194 cpu.go:275] no items to output this cycle E1121 20:14:56.413666 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:14:56.413701 3798194 memory.go:191] Add success. W1121 20:14:56.413733 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:14:56.413745 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:14:56.413748 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:14:56.418738 3798194 cpu.go:282] Add success. I1121 20:14:56.441274 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:14:56.441782 3798194 net.go:642] Add success. I1121 20:14:56.443538 3798194 net.go:778] primary dev: ETH0 I1121 20:14:56.443561 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:14:56.443576 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:14:57.445485 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:14:57.445754 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:14:57.445766 3798194 disk_worker.go:723] disk space is not compliant W1121 20:14:57.445769 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:14:57.447889 3798194 disk_worker.go:509] system disk:vda1 I1121 20:14:57.447926 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:14:58.446369 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:14:59.444365 3798194 disk_info.go:125] begin check local disk info of client I1121 20:14:59.448209 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:14:59.448218 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004100c0 0xc000410100] I1121 20:14:59.450344 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:14:59.450401 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:14:59.450424 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:14:59.470930 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:15:06.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:15:06.412877 3798194 memory.go:184] no items to output this cycle I1121 20:15:06.413028 3798194 cpu.go:275] no items to output this cycle E1121 20:15:16.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:15:16.412266 3798194 memory.go:184] no items to output this cycle I1121 20:15:16.412424 3798194 cpu.go:275] no items to output this cycle I1121 20:15:17.550755 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:15:17.550764 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:15:26.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:15:26.412270 3798194 memory.go:191] Add success. I1121 20:15:26.416662 3798194 cpu.go:282] Add success. I1121 20:15:26.441273 3798194 net.go:642] Add success. I1121 20:15:26.442815 3798194 net.go:778] primary dev: ETH0 I1121 20:15:26.442832 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:15:26.442846 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:15:29.449945 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:15:29.450022 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:15:29.450049 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:15:36.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:15:36.413268 3798194 memory.go:184] no items to output this cycle I1121 20:15:36.413408 3798194 cpu.go:275] no items to output this cycle E1121 20:15:46.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:15:46.412880 3798194 memory.go:184] no items to output this cycle I1121 20:15:46.413028 3798194 cpu.go:275] no items to output this cycle E1121 20:15:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:15:56.411949 3798194 memory.go:191] Add success. W1121 20:15:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:15:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:15:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:15:56.412353 3798194 cpu.go:282] Add success. I1121 20:15:56.441570 3798194 event_worker.go:152] Polling the log file for events... I1121 20:15:56.444678 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:15:56.445183 3798194 net.go:642] Add success. I1121 20:15:56.450009 3798194 net.go:778] primary dev: ETH0 I1121 20:15:56.450023 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:15:56.450038 3798194 net.go:659] Add eni metrics success, count: 7 E1121 20:15:57.450614 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 20:15:57.450628 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:15:57.450829 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:15:57.450839 3798194 disk_worker.go:723] disk space is not compliant W1121 20:15:57.450842 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:15:57.459138 3798194 disk_worker.go:509] system disk:vda1 I1121 20:15:57.459179 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 20:15:58.448079 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 20:15:58.448093 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:15:59.448578 3798194 disk_info.go:125] begin check local disk info of client I1121 20:15:59.452181 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:15:59.452231 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:15:59.452249 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:15:59.452370 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 20:15:59.452384 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:15:59.452388 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a1800 0xc0003a1840] I1121 20:15:59.470725 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:16:06.412962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:16:06.412984 3798194 memory.go:184] no items to output this cycle I1121 20:16:06.413138 3798194 cpu.go:275] no items to output this cycle E1121 20:16:16.412471 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:16:16.412493 3798194 memory.go:184] no items to output this cycle I1121 20:16:16.412636 3798194 cpu.go:275] no items to output this cycle E1121 20:16:26.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:16:26.411948 3798194 memory.go:191] Add success. I1121 20:16:26.412320 3798194 cpu.go:282] Add success. I1121 20:16:26.442024 3798194 net.go:642] Add success. I1121 20:16:26.446637 3798194 net.go:778] primary dev: ETH0 I1121 20:16:26.446760 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:16:26.446776 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:16:29.450191 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:16:29.450265 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:16:29.450292 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:16:36.412958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:16:36.412981 3798194 memory.go:184] no items to output this cycle I1121 20:16:36.413143 3798194 cpu.go:275] no items to output this cycle E1121 20:16:46.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:16:46.412886 3798194 memory.go:184] no items to output this cycle I1121 20:16:46.413036 3798194 cpu.go:275] no items to output this cycle E1121 20:16:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:16:56.411950 3798194 memory.go:191] Add success. W1121 20:16:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:16:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:16:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:16:56.412338 3798194 cpu.go:282] Add success. I1121 20:16:56.441798 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:16:56.442302 3798194 net.go:642] Add success. I1121 20:16:56.446427 3798194 net.go:778] primary dev: ETH0 I1121 20:16:56.446442 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:16:56.446456 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:16:57.450592 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:16:57.450799 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:16:57.450810 3798194 disk_worker.go:723] disk space is not compliant W1121 20:16:57.450814 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:16:57.459124 3798194 disk_worker.go:509] system disk:vda1 I1121 20:16:57.459168 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:16:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:16:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:16:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:16:59.450133 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:16:59.452615 3798194 disk_info.go:125] begin check local disk info of client I1121 20:16:59.456456 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:16:59.456463 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000403cc0 0xc000403d00] I1121 20:16:59.471612 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:17:06.413170 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:17:06.413197 3798194 memory.go:184] no items to output this cycle I1121 20:17:06.413341 3798194 cpu.go:275] no items to output this cycle E1121 20:17:16.412325 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:17:16.412347 3798194 memory.go:184] no items to output this cycle I1121 20:17:16.412497 3798194 cpu.go:275] no items to output this cycle E1121 20:17:26.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:17:26.411969 3798194 memory.go:191] Add success. I1121 20:17:26.412307 3798194 cpu.go:282] Add success. I1121 20:17:26.441884 3798194 net.go:642] Add success. I1121 20:17:26.446093 3798194 net.go:778] primary dev: ETH0 I1121 20:17:26.446127 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:17:26.446141 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:17:29.450257 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:17:29.450333 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:17:29.450360 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:17:36.412370 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:17:36.412392 3798194 memory.go:184] no items to output this cycle I1121 20:17:36.412531 3798194 cpu.go:275] no items to output this cycle E1121 20:17:46.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:17:46.413254 3798194 memory.go:184] no items to output this cycle I1121 20:17:46.413399 3798194 cpu.go:275] no items to output this cycle E1121 20:17:56.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:17:56.413259 3798194 memory.go:191] Add success. W1121 20:17:56.413294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:17:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:17:56.413309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:17:56.417985 3798194 cpu.go:282] Add success. I1121 20:17:56.442074 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:17:56.443023 3798194 net.go:642] Add success. I1121 20:17:56.445174 3798194 net.go:778] primary dev: ETH0 I1121 20:17:56.445190 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:17:56.445202 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:17:57.449552 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:17:57.449805 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:17:57.449819 3798194 disk_worker.go:723] disk space is not compliant W1121 20:17:57.449822 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:17:57.451631 3798194 disk_worker.go:509] system disk:vda1 I1121 20:17:57.451670 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:17:58.446383 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:17:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:17:59.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:17:59.450506 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:17:59.457005 3798194 disk_info.go:125] begin check local disk info of client I1121 20:17:59.460942 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:17:59.460950 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004c39c0 0xc0004c3a00] I1121 20:17:59.471079 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:18:06.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:18:06.412269 3798194 memory.go:184] no items to output this cycle I1121 20:18:06.412415 3798194 cpu.go:275] no items to output this cycle E1121 20:18:16.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:18:16.413271 3798194 memory.go:184] no items to output this cycle I1121 20:18:16.413424 3798194 cpu.go:275] no items to output this cycle I1121 20:18:17.550934 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:18:17.550943 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:18:26.412366 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:18:26.412403 3798194 memory.go:191] Add success. I1121 20:18:26.416908 3798194 cpu.go:282] Add success. I1121 20:18:26.441688 3798194 net.go:642] Add success. I1121 20:18:26.443182 3798194 net.go:778] primary dev: ETH0 I1121 20:18:26.443197 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:18:26.443208 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:18:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:18:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:18:29.450496 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:18:36.411751 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:18:36.411774 3798194 memory.go:184] no items to output this cycle I1121 20:18:36.411922 3798194 cpu.go:275] no items to output this cycle E1121 20:18:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:18:46.411982 3798194 memory.go:184] no items to output this cycle I1121 20:18:46.412139 3798194 cpu.go:275] no items to output this cycle E1121 20:18:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:18:56.413264 3798194 memory.go:191] Add success. W1121 20:18:56.413298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:18:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:18:56.413315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:18:56.417677 3798194 cpu.go:282] Add success. I1121 20:18:56.441217 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:18:56.441696 3798194 net.go:642] Add success. I1121 20:18:56.443366 3798194 net.go:778] primary dev: ETH0 I1121 20:18:56.443380 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:18:56.443394 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:18:57.451036 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:18:57.451266 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:18:57.451277 3798194 disk_worker.go:723] disk space is not compliant W1121 20:18:57.451281 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:18:57.460094 3798194 disk_worker.go:509] system disk:vda1 I1121 20:18:57.460160 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:18:58.446364 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:18:59.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:18:59.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:18:59.450502 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:18:59.461971 3798194 disk_info.go:125] begin check local disk info of client I1121 20:18:59.465848 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:18:59.465856 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004041c0 0xc000404200] I1121 20:18:59.470985 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:19:06.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:19:06.413265 3798194 memory.go:184] no items to output this cycle I1121 20:19:06.413410 3798194 cpu.go:275] no items to output this cycle E1121 20:19:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:19:16.413259 3798194 memory.go:184] no items to output this cycle I1121 20:19:16.413424 3798194 cpu.go:275] no items to output this cycle E1121 20:19:26.413224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:19:26.413261 3798194 memory.go:191] Add success. I1121 20:19:26.418061 3798194 cpu.go:282] Add success. I1121 20:19:26.441577 3798194 net.go:642] Add success. I1121 20:19:26.443037 3798194 net.go:778] primary dev: ETH0 I1121 20:19:26.443053 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:19:26.443067 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:19:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:19:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:19:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:19:36.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:19:36.413253 3798194 memory.go:184] no items to output this cycle I1121 20:19:36.413406 3798194 cpu.go:275] no items to output this cycle E1121 20:19:46.413138 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:19:46.413166 3798194 memory.go:184] no items to output this cycle I1121 20:19:46.413324 3798194 cpu.go:275] no items to output this cycle E1121 20:19:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:19:56.412265 3798194 memory.go:191] Add success. W1121 20:19:56.412301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:19:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:19:56.412316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:19:56.419638 3798194 cpu.go:282] Add success. I1121 20:19:56.440948 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:19:56.441524 3798194 net.go:642] Add success. I1121 20:19:56.443249 3798194 net.go:778] primary dev: ETH0 I1121 20:19:56.443264 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:19:56.443276 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:19:57.446425 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:19:57.446697 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:19:57.446710 3798194 disk_worker.go:723] disk space is not compliant W1121 20:19:57.446714 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:19:57.449529 3798194 disk_worker.go:509] system disk:vda1 I1121 20:19:57.449568 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:19:58.446381 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:19:59.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:19:59.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:19:59.450495 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:19:59.465932 3798194 disk_info.go:125] begin check local disk info of client I1121 20:19:59.469695 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:19:59.469703 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00038b840 0xc00038b880] I1121 20:19:59.470773 3798194 disk_local_worker.go:436] Get disk info: [] E1121 20:20:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:20:06.412261 3798194 memory.go:184] no items to output this cycle I1121 20:20:06.412402 3798194 cpu.go:275] no items to output this cycle E1121 20:20:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:20:16.411988 3798194 memory.go:184] no items to output this cycle I1121 20:20:16.412149 3798194 cpu.go:275] no items to output this cycle E1121 20:20:26.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:20:26.411946 3798194 memory.go:191] Add success. I1121 20:20:26.412493 3798194 cpu.go:282] Add success. I1121 20:20:26.441300 3798194 net.go:642] Add success. I1121 20:20:26.445464 3798194 net.go:778] primary dev: ETH0 I1121 20:20:26.445478 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:20:26.445491 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:20:29.450007 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:20:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:20:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:20:36.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:20:36.412885 3798194 memory.go:184] no items to output this cycle I1121 20:20:36.413028 3798194 cpu.go:275] no items to output this cycle E1121 20:20:46.411945 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:20:46.411967 3798194 memory.go:184] no items to output this cycle I1121 20:20:46.412122 3798194 cpu.go:275] no items to output this cycle E1121 20:20:56.411233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:20:56.411266 3798194 memory.go:191] Add success. W1121 20:20:56.411294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:20:56.411305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:20:56.411308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:20:56.415811 3798194 cpu.go:282] Add success. I1121 20:20:56.444057 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:20:56.444645 3798194 net.go:642] Add success. I1121 20:20:56.446418 3798194 net.go:778] primary dev: ETH0 I1121 20:20:56.446432 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:20:56.446444 3798194 net.go:659] Add eni metrics success, count: 7 E1121 20:20:57.450617 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 20:20:57.450630 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:20:57.450781 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:20:57.450832 3798194 disk_worker.go:723] disk space is not compliant W1121 20:20:57.450835 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:20:57.459195 3798194 disk_worker.go:509] system disk:vda1 I1121 20:20:57.459237 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 20:20:58.447994 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 20:20:58.448009 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 20:20:59.448996 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 20:20:59.450071 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:20:59.450132 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:20:59.450150 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:20:59.470670 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:20:59.470697 3798194 disk_info.go:125] begin check local disk info of client I1121 20:20:59.474408 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:20:59.474415 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003dbe40 0xc0003dbe80] E1121 20:21:06.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:21:06.411986 3798194 memory.go:184] no items to output this cycle I1121 20:21:06.412154 3798194 cpu.go:275] no items to output this cycle E1121 20:21:16.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:21:16.412255 3798194 memory.go:184] no items to output this cycle I1121 20:21:16.412401 3798194 cpu.go:275] no items to output this cycle I1121 20:21:17.551076 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:21:17.551085 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:21:26.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:21:26.412267 3798194 memory.go:191] Add success. I1121 20:21:26.416670 3798194 cpu.go:282] Add success. I1121 20:21:26.441261 3798194 net.go:642] Add success. I1121 20:21:26.442917 3798194 net.go:778] primary dev: ETH0 I1121 20:21:26.442933 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:21:26.442947 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:21:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:21:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:21:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:21:36.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:21:36.412870 3798194 memory.go:184] no items to output this cycle I1121 20:21:36.413017 3798194 cpu.go:275] no items to output this cycle E1121 20:21:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:21:46.412252 3798194 memory.go:184] no items to output this cycle I1121 20:21:46.412393 3798194 cpu.go:275] no items to output this cycle E1121 20:21:56.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:21:56.413270 3798194 memory.go:191] Add success. W1121 20:21:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:21:56.413315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:21:56.413319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:21:56.417726 3798194 cpu.go:282] Add success. I1121 20:21:56.441272 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:21:56.441824 3798194 net.go:642] Add success. I1121 20:21:56.443336 3798194 net.go:778] primary dev: ETH0 I1121 20:21:56.443351 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:21:56.443364 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:21:57.450998 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:21:57.451229 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:21:57.451240 3798194 disk_worker.go:723] disk space is not compliant W1121 20:21:57.451243 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:21:57.459990 3798194 disk_worker.go:509] system disk:vda1 I1121 20:21:57.460030 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:21:58.446371 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:21:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:21:59.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:21:59.450492 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:21:59.471009 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:21:59.475128 3798194 disk_info.go:125] begin check local disk info of client I1121 20:21:59.478883 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:21:59.478894 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030f940 0xc00030f980] E1121 20:22:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:22:06.412253 3798194 memory.go:184] no items to output this cycle I1121 20:22:06.412396 3798194 cpu.go:275] no items to output this cycle E1121 20:22:16.412256 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:22:16.412280 3798194 memory.go:184] no items to output this cycle I1121 20:22:16.412440 3798194 cpu.go:275] no items to output this cycle E1121 20:22:26.413047 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:22:26.413080 3798194 memory.go:191] Add success. I1121 20:22:26.417421 3798194 cpu.go:282] Add success. I1121 20:22:26.442089 3798194 net.go:642] Add success. I1121 20:22:26.443624 3798194 net.go:778] primary dev: ETH0 I1121 20:22:26.443638 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:22:26.443651 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:22:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:22:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:22:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:22:36.412000 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:22:36.412022 3798194 memory.go:184] no items to output this cycle I1121 20:22:36.412187 3798194 cpu.go:275] no items to output this cycle E1121 20:22:46.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:22:46.412255 3798194 memory.go:184] no items to output this cycle I1121 20:22:46.412402 3798194 cpu.go:275] no items to output this cycle E1121 20:22:56.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:22:56.412279 3798194 memory.go:191] Add success. W1121 20:22:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:22:56.412322 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:22:56.412326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:22:56.416873 3798194 cpu.go:282] Add success. I1121 20:22:56.441362 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:22:56.441929 3798194 net.go:642] Add success. I1121 20:22:56.444163 3798194 net.go:778] primary dev: ETH0 I1121 20:22:56.444179 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:22:56.444193 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:22:57.452162 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:22:57.452335 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:22:57.452391 3798194 disk_worker.go:723] disk space is not compliant W1121 20:22:57.452395 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:22:57.461923 3798194 disk_worker.go:509] system disk:vda1 I1121 20:22:57.461965 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:22:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:22:59.450614 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:22:59.450694 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:22:59.450719 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:22:59.471187 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:22:59.479331 3798194 disk_info.go:125] begin check local disk info of client I1121 20:22:59.483288 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:22:59.483297 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000457900 0xc000457940] E1121 20:23:06.412831 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:23:06.412854 3798194 memory.go:184] no items to output this cycle I1121 20:23:06.413007 3798194 cpu.go:275] no items to output this cycle E1121 20:23:16.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:23:16.412260 3798194 memory.go:184] no items to output this cycle I1121 20:23:16.412417 3798194 cpu.go:275] no items to output this cycle E1121 20:23:26.412282 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:23:26.412327 3798194 memory.go:191] Add success. I1121 20:23:26.419382 3798194 cpu.go:282] Add success. I1121 20:23:26.442508 3798194 net.go:642] Add success. I1121 20:23:26.443930 3798194 net.go:778] primary dev: ETH0 I1121 20:23:26.443948 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:23:26.443962 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:23:29.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:23:29.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:23:29.450151 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:23:36.412844 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:23:36.412866 3798194 memory.go:184] no items to output this cycle I1121 20:23:36.413009 3798194 cpu.go:275] no items to output this cycle E1121 20:23:46.412767 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:23:46.412793 3798194 memory.go:184] no items to output this cycle I1121 20:23:46.412939 3798194 cpu.go:275] no items to output this cycle E1121 20:23:56.411900 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:23:56.411933 3798194 memory.go:191] Add success. W1121 20:23:56.411961 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:23:56.411972 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:23:56.411976 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:23:56.412325 3798194 cpu.go:282] Add success. I1121 20:23:56.441833 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:23:56.442363 3798194 net.go:642] Add success. I1121 20:23:56.446613 3798194 net.go:778] primary dev: ETH0 I1121 20:23:56.446628 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:23:56.446639 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:23:57.451011 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:23:57.451246 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:23:57.451257 3798194 disk_worker.go:723] disk space is not compliant W1121 20:23:57.451261 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:23:57.460009 3798194 disk_worker.go:509] system disk:vda1 I1121 20:23:57.460054 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:23:58.446454 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:23:59.450422 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:23:59.450497 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:23:59.450534 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:23:59.471210 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:23:59.487145 3798194 disk_info.go:125] begin check local disk info of client I1121 20:23:59.492664 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:23:59.492674 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b8b80 0xc0003b8bc0] E1121 20:24:06.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:24:06.413261 3798194 memory.go:184] no items to output this cycle I1121 20:24:06.413412 3798194 cpu.go:275] no items to output this cycle E1121 20:24:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:24:16.412257 3798194 memory.go:184] no items to output this cycle I1121 20:24:16.412405 3798194 cpu.go:275] no items to output this cycle I1121 20:24:17.551225 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:24:17.551233 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:24:26.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:24:26.412274 3798194 memory.go:191] Add success. I1121 20:24:26.416594 3798194 cpu.go:282] Add success. I1121 20:24:26.442155 3798194 net.go:642] Add success. I1121 20:24:26.444157 3798194 net.go:778] primary dev: ETH0 I1121 20:24:26.444181 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:24:26.444200 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:24:29.449668 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:24:29.449749 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:24:29.449777 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:24:36.413038 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:24:36.413060 3798194 memory.go:184] no items to output this cycle I1121 20:24:36.413226 3798194 cpu.go:275] no items to output this cycle E1121 20:24:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:24:46.412253 3798194 memory.go:184] no items to output this cycle I1121 20:24:46.412398 3798194 cpu.go:275] no items to output this cycle E1121 20:24:56.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:24:56.412262 3798194 memory.go:191] Add success. W1121 20:24:56.412289 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:24:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:24:56.412304 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:24:56.416668 3798194 cpu.go:282] Add success. I1121 20:24:56.440948 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:24:56.441528 3798194 net.go:642] Add success. I1121 20:24:56.443052 3798194 net.go:778] primary dev: ETH0 I1121 20:24:56.443066 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:24:56.443077 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:24:57.450582 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:24:57.450781 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:24:57.450792 3798194 disk_worker.go:723] disk space is not compliant W1121 20:24:57.450795 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:24:57.459149 3798194 disk_worker.go:509] system disk:vda1 I1121 20:24:57.459188 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:24:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:24:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:24:59.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:24:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:24:59.471686 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:24:59.492886 3798194 disk_info.go:125] begin check local disk info of client I1121 20:24:59.496707 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:24:59.496715 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000301400 0xc000301440] E1121 20:25:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:25:06.412262 3798194 memory.go:184] no items to output this cycle I1121 20:25:06.412402 3798194 cpu.go:275] no items to output this cycle E1121 20:25:16.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:25:16.412265 3798194 memory.go:184] no items to output this cycle I1121 20:25:16.412409 3798194 cpu.go:275] no items to output this cycle E1121 20:25:26.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:25:26.412887 3798194 memory.go:191] Add success. I1121 20:25:26.417314 3798194 cpu.go:282] Add success. I1121 20:25:26.441740 3798194 net.go:642] Add success. I1121 20:25:26.443292 3798194 net.go:778] primary dev: ETH0 I1121 20:25:26.443306 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:25:26.443320 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:25:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:25:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:25:29.450495 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:25:36.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:25:36.412883 3798194 memory.go:184] no items to output this cycle I1121 20:25:36.413046 3798194 cpu.go:275] no items to output this cycle E1121 20:25:46.412159 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:25:46.412187 3798194 memory.go:184] no items to output this cycle I1121 20:25:46.412332 3798194 cpu.go:275] no items to output this cycle E1121 20:25:56.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:25:56.412261 3798194 memory.go:191] Add success. W1121 20:25:56.412288 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:25:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:25:56.412303 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:25:56.416817 3798194 cpu.go:282] Add success. I1121 20:25:56.443861 3798194 event_worker.go:152] Polling the log file for events... I1121 20:25:56.444069 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:25:56.444550 3798194 net.go:642] Add success. I1121 20:25:56.445982 3798194 net.go:778] primary dev: ETH0 I1121 20:25:56.445996 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:25:56.446008 3798194 net.go:659] Add eni metrics success, count: 7 E1121 20:25:57.450619 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 20:25:57.450632 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:25:57.450801 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:25:57.450852 3798194 disk_worker.go:723] disk space is not compliant W1121 20:25:57.450855 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:25:57.459273 3798194 disk_worker.go:509] system disk:vda1 I1121 20:25:57.459313 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 20:25:58.448097 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 20:25:58.448132 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 20:25:59.448612 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 20:25:59.449686 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:25:59.449737 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:25:59.449756 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:25:59.471260 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:25:59.497476 3798194 disk_info.go:125] begin check local disk info of client I1121 20:25:59.501153 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:25:59.501160 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e4f00 0xc0002e4f40] E1121 20:26:06.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:26:06.411987 3798194 memory.go:184] no items to output this cycle I1121 20:26:06.412148 3798194 cpu.go:275] no items to output this cycle E1121 20:26:16.413046 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:26:16.413070 3798194 memory.go:184] no items to output this cycle I1121 20:26:16.413238 3798194 cpu.go:275] no items to output this cycle E1121 20:26:26.412048 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:26:26.412086 3798194 memory.go:191] Add success. I1121 20:26:26.416526 3798194 cpu.go:282] Add success. I1121 20:26:26.442072 3798194 net.go:642] Add success. I1121 20:26:26.443627 3798194 net.go:778] primary dev: ETH0 I1121 20:26:26.443643 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:26:26.443661 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:26:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:26:29.450466 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:26:29.450492 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:26:36.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:26:36.412248 3798194 memory.go:184] no items to output this cycle I1121 20:26:36.412395 3798194 cpu.go:275] no items to output this cycle E1121 20:26:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:26:46.413259 3798194 memory.go:184] no items to output this cycle I1121 20:26:46.413408 3798194 cpu.go:275] no items to output this cycle E1121 20:26:56.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:26:56.412891 3798194 memory.go:191] Add success. W1121 20:26:56.412927 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:26:56.412939 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:26:56.412942 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:26:56.417679 3798194 cpu.go:282] Add success. I1121 20:26:56.440960 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:26:56.441529 3798194 net.go:642] Add success. I1121 20:26:56.443184 3798194 net.go:778] primary dev: ETH0 I1121 20:26:56.443198 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:26:56.443211 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:26:57.450932 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:26:57.451088 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:26:57.451163 3798194 disk_worker.go:723] disk space is not compliant W1121 20:26:57.451166 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:26:57.460665 3798194 disk_worker.go:509] system disk:vda1 I1121 20:26:57.460706 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:26:58.446605 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:26:59.449658 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:26:59.449736 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:26:59.449761 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:26:59.471290 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:26:59.501531 3798194 disk_info.go:125] begin check local disk info of client I1121 20:26:59.505347 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:26:59.505355 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047d680 0xc00047d6c0] E1121 20:27:06.412989 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:27:06.413013 3798194 memory.go:184] no items to output this cycle I1121 20:27:06.413179 3798194 cpu.go:275] no items to output this cycle E1121 20:27:16.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:27:16.413255 3798194 memory.go:184] no items to output this cycle I1121 20:27:16.413407 3798194 cpu.go:275] no items to output this cycle I1121 20:27:17.551392 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:27:17.551401 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:27:26.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:27:26.413263 3798194 memory.go:191] Add success. I1121 20:27:26.417318 3798194 cpu.go:282] Add success. I1121 20:27:26.441738 3798194 net.go:642] Add success. I1121 20:27:26.451633 3798194 net.go:778] primary dev: ETH0 I1121 20:27:26.451648 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:27:26.451662 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:27:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:27:29.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:27:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:27:36.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:27:36.412883 3798194 memory.go:184] no items to output this cycle I1121 20:27:36.413035 3798194 cpu.go:275] no items to output this cycle E1121 20:27:46.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:27:46.413262 3798194 memory.go:184] no items to output this cycle I1121 20:27:46.413418 3798194 cpu.go:275] no items to output this cycle E1121 20:27:56.412038 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:27:56.412082 3798194 memory.go:191] Add success. W1121 20:27:56.412129 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:27:56.412147 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:27:56.412154 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:27:56.417342 3798194 cpu.go:282] Add success. I1121 20:27:56.441738 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:27:56.442245 3798194 net.go:642] Add success. I1121 20:27:56.444001 3798194 net.go:778] primary dev: ETH0 I1121 20:27:56.444015 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:27:56.444027 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:27:57.450572 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:27:57.450736 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:27:57.450803 3798194 disk_worker.go:723] disk space is not compliant W1121 20:27:57.450807 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:27:57.459087 3798194 disk_worker.go:509] system disk:vda1 I1121 20:27:57.459142 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:27:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:27:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:27:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:27:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:27:59.471665 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:27:59.505913 3798194 disk_info.go:125] begin check local disk info of client I1121 20:27:59.509715 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:27:59.509723 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f6340 0xc0003f6380] E1121 20:28:06.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:28:06.412268 3798194 memory.go:184] no items to output this cycle I1121 20:28:06.412410 3798194 cpu.go:275] no items to output this cycle E1121 20:28:16.413093 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:28:16.413129 3798194 memory.go:184] no items to output this cycle I1121 20:28:16.413279 3798194 cpu.go:275] no items to output this cycle E1121 20:28:26.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:28:26.412883 3798194 memory.go:191] Add success. I1121 20:28:26.417198 3798194 cpu.go:282] Add success. I1121 20:28:26.441870 3798194 net.go:642] Add success. I1121 20:28:26.443428 3798194 net.go:778] primary dev: ETH0 I1121 20:28:26.443444 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:28:26.443462 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:28:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:28:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:28:29.450129 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:28:36.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:28:36.411971 3798194 memory.go:184] no items to output this cycle I1121 20:28:36.412131 3798194 cpu.go:275] no items to output this cycle E1121 20:28:46.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:28:46.412256 3798194 memory.go:184] no items to output this cycle I1121 20:28:46.412397 3798194 cpu.go:275] no items to output this cycle E1121 20:28:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:28:56.411952 3798194 memory.go:191] Add success. W1121 20:28:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:28:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:28:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:28:56.412353 3798194 cpu.go:282] Add success. I1121 20:28:56.441915 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:28:56.442471 3798194 net.go:642] Add success. I1121 20:28:56.446399 3798194 net.go:778] primary dev: ETH0 I1121 20:28:56.446415 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:28:56.446428 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:28:57.450709 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:28:57.450871 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:28:57.450924 3798194 disk_worker.go:723] disk space is not compliant W1121 20:28:57.450928 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:28:57.459265 3798194 disk_worker.go:509] system disk:vda1 I1121 20:28:57.459312 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:28:58.445699 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:28:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:28:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:28:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:28:59.471650 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:28:59.509804 3798194 disk_info.go:125] begin check local disk info of client I1121 20:28:59.513612 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:28:59.513619 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00048c780 0xc00048c7c0] E1121 20:29:06.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:29:06.413264 3798194 memory.go:184] no items to output this cycle I1121 20:29:06.413412 3798194 cpu.go:275] no items to output this cycle E1121 20:29:16.412328 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:29:16.412351 3798194 memory.go:184] no items to output this cycle I1121 20:29:16.412497 3798194 cpu.go:275] no items to output this cycle E1121 20:29:26.412045 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:29:26.412079 3798194 memory.go:191] Add success. I1121 20:29:26.416962 3798194 cpu.go:282] Add success. I1121 20:29:26.441484 3798194 net.go:642] Add success. I1121 20:29:26.442974 3798194 net.go:778] primary dev: ETH0 I1121 20:29:26.442989 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:29:26.443003 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:29:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:29:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:29:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:29:36.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:29:36.411980 3798194 memory.go:184] no items to output this cycle I1121 20:29:36.412127 3798194 cpu.go:275] no items to output this cycle E1121 20:29:46.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:29:46.413253 3798194 memory.go:184] no items to output this cycle I1121 20:29:46.413405 3798194 cpu.go:275] no items to output this cycle E1121 20:29:56.412825 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:29:56.412859 3798194 memory.go:191] Add success. W1121 20:29:56.412885 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:29:56.412897 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:29:56.412900 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:29:56.417442 3798194 cpu.go:282] Add success. I1121 20:29:56.441834 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:29:56.442631 3798194 net.go:642] Add success. I1121 20:29:56.444239 3798194 net.go:778] primary dev: ETH0 I1121 20:29:56.444254 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:29:56.444268 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:29:57.451100 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:29:57.451319 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:29:57.451332 3798194 disk_worker.go:723] disk space is not compliant W1121 20:29:57.451334 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:29:57.460067 3798194 disk_worker.go:509] system disk:vda1 I1121 20:29:57.460106 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:29:58.446047 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:29:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:29:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:29:59.450140 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:29:59.471674 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:29:59.513973 3798194 disk_info.go:125] begin check local disk info of client I1121 20:29:59.517792 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:29:59.517799 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004f79c0 0xc0004f7a00] E1121 20:30:06.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:30:06.412885 3798194 memory.go:184] no items to output this cycle I1121 20:30:06.413029 3798194 cpu.go:275] no items to output this cycle E1121 20:30:16.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:30:16.413254 3798194 memory.go:184] no items to output this cycle I1121 20:30:16.413401 3798194 cpu.go:275] no items to output this cycle I1121 20:30:17.551569 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:30:17.551577 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:30:26.411221 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:30:26.411254 3798194 memory.go:191] Add success. I1121 20:30:26.411594 3798194 cpu.go:282] Add success. I1121 20:30:26.442278 3798194 net.go:642] Add success. I1121 20:30:26.446481 3798194 net.go:778] primary dev: ETH0 I1121 20:30:26.446496 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:30:26.446508 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:30:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:30:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:30:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:30:36.412802 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:30:36.412825 3798194 memory.go:184] no items to output this cycle I1121 20:30:36.412966 3798194 cpu.go:275] no items to output this cycle E1121 20:30:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:30:46.413262 3798194 memory.go:184] no items to output this cycle I1121 20:30:46.413407 3798194 cpu.go:275] no items to output this cycle E1121 20:30:56.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:30:56.413260 3798194 memory.go:191] Add success. W1121 20:30:56.413285 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:30:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:30:56.413300 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:30:56.417468 3798194 cpu.go:282] Add success. I1121 20:30:56.443810 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:30:56.444337 3798194 net.go:642] Add success. I1121 20:30:56.445830 3798194 net.go:778] primary dev: ETH0 I1121 20:30:56.445845 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:30:56.445857 3798194 net.go:659] Add eni metrics success, count: 7 E1121 20:30:57.450670 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 20:30:57.450683 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:30:57.450829 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:30:57.450878 3798194 disk_worker.go:723] disk space is not compliant W1121 20:30:57.450881 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:30:57.459190 3798194 disk_worker.go:509] system disk:vda1 I1121 20:30:57.459236 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 20:30:58.448103 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 20:30:58.448132 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 20:30:59.448595 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 20:30:59.449670 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:30:59.449721 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:30:59.449740 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:30:59.471251 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:30:59.518567 3798194 disk_info.go:125] begin check local disk info of client I1121 20:30:59.522236 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:30:59.522244 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040ea00 0xc00040ea40] E1121 20:31:06.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:31:06.413253 3798194 memory.go:184] no items to output this cycle I1121 20:31:06.413396 3798194 cpu.go:275] no items to output this cycle E1121 20:31:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:31:16.412256 3798194 memory.go:184] no items to output this cycle I1121 20:31:16.412645 3798194 cpu.go:275] no items to output this cycle E1121 20:31:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:31:26.411950 3798194 memory.go:191] Add success. I1121 20:31:26.412340 3798194 cpu.go:282] Add success. I1121 20:31:26.442032 3798194 net.go:642] Add success. I1121 20:31:26.445606 3798194 net.go:778] primary dev: ETH0 I1121 20:31:26.445621 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:31:26.445634 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:31:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:31:29.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:31:29.450512 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:31:36.413251 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:31:36.413273 3798194 memory.go:184] no items to output this cycle I1121 20:31:36.413421 3798194 cpu.go:275] no items to output this cycle E1121 20:31:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:31:46.411985 3798194 memory.go:184] no items to output this cycle I1121 20:31:46.412148 3798194 cpu.go:275] no items to output this cycle E1121 20:31:56.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:31:56.411942 3798194 memory.go:191] Add success. W1121 20:31:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:31:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:31:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:31:56.412348 3798194 cpu.go:282] Add success. I1121 20:31:56.441812 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:31:56.442320 3798194 net.go:642] Add success. I1121 20:31:56.446233 3798194 net.go:778] primary dev: ETH0 I1121 20:31:56.446248 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:31:56.446262 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:31:57.451007 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:31:57.451203 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:31:57.451257 3798194 disk_worker.go:723] disk space is not compliant W1121 20:31:57.451260 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:31:57.463670 3798194 disk_worker.go:509] system disk:vda1 I1121 20:31:57.463711 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:31:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:31:59.450105 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:31:59.450205 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:31:59.450243 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:31:59.470726 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:31:59.523082 3798194 disk_info.go:125] begin check local disk info of client I1121 20:31:59.526911 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:31:59.526919 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000313c40 0xc000313c80] E1121 20:32:06.412780 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:32:06.412808 3798194 memory.go:184] no items to output this cycle I1121 20:32:06.412968 3798194 cpu.go:275] no items to output this cycle E1121 20:32:16.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:32:16.412259 3798194 memory.go:184] no items to output this cycle I1121 20:32:16.412413 3798194 cpu.go:275] no items to output this cycle E1121 20:32:26.411909 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:32:26.411941 3798194 memory.go:191] Add success. I1121 20:32:26.412290 3798194 cpu.go:282] Add success. I1121 20:32:26.441767 3798194 net.go:642] Add success. I1121 20:32:26.445991 3798194 net.go:778] primary dev: ETH0 I1121 20:32:26.446006 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:32:26.446019 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:32:29.449668 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:32:29.449753 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:32:29.449781 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:32:36.412903 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:32:36.412926 3798194 memory.go:184] no items to output this cycle I1121 20:32:36.413067 3798194 cpu.go:275] no items to output this cycle E1121 20:32:46.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:32:46.412266 3798194 memory.go:184] no items to output this cycle I1121 20:32:46.412415 3798194 cpu.go:275] no items to output this cycle E1121 20:32:56.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:32:56.413282 3798194 memory.go:191] Add success. W1121 20:32:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:32:56.413323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:32:56.413327 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:32:56.417502 3798194 cpu.go:282] Add success. I1121 20:32:56.441891 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:32:56.442505 3798194 net.go:642] Add success. I1121 20:32:56.443896 3798194 net.go:778] primary dev: ETH0 I1121 20:32:56.443911 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:32:56.443923 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:32:57.451295 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:32:57.451495 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:32:57.451506 3798194 disk_worker.go:723] disk space is not compliant W1121 20:32:57.451509 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:32:57.459878 3798194 disk_worker.go:509] system disk:vda1 I1121 20:32:57.459934 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:32:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:32:59.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:32:59.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:32:59.450512 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:32:59.471051 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:32:59.527196 3798194 disk_info.go:125] begin check local disk info of client I1121 20:32:59.530955 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:32:59.530965 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e1580 0xc0002e15c0] E1121 20:33:06.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:33:06.412880 3798194 memory.go:184] no items to output this cycle I1121 20:33:06.413027 3798194 cpu.go:275] no items to output this cycle E1121 20:33:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:33:16.411974 3798194 memory.go:184] no items to output this cycle I1121 20:33:16.412136 3798194 cpu.go:275] no items to output this cycle I1121 20:33:17.552271 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:33:17.552279 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:33:26.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:33:26.413270 3798194 memory.go:191] Add success. I1121 20:33:26.417348 3798194 cpu.go:282] Add success. I1121 20:33:26.441745 3798194 net.go:642] Add success. I1121 20:33:26.443218 3798194 net.go:778] primary dev: ETH0 I1121 20:33:26.443233 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:33:26.443246 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:33:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:33:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:33:29.450516 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:33:36.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:33:36.413258 3798194 memory.go:184] no items to output this cycle I1121 20:33:36.413401 3798194 cpu.go:275] no items to output this cycle E1121 20:33:46.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:33:46.412258 3798194 memory.go:184] no items to output this cycle I1121 20:33:46.412407 3798194 cpu.go:275] no items to output this cycle E1121 20:33:56.412841 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:33:56.412875 3798194 memory.go:191] Add success. W1121 20:33:56.412912 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:33:56.412923 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:33:56.412926 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:33:56.416872 3798194 cpu.go:282] Add success. I1121 20:33:56.441294 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:33:56.441786 3798194 net.go:642] Add success. I1121 20:33:56.443197 3798194 net.go:778] primary dev: ETH0 I1121 20:33:56.443212 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:33:56.443226 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:33:57.450807 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:33:57.451017 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:33:57.451028 3798194 disk_worker.go:723] disk space is not compliant W1121 20:33:57.451031 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:33:57.459784 3798194 disk_worker.go:509] system disk:vda1 I1121 20:33:57.459826 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:33:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:33:59.450441 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:33:59.450524 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:33:59.450564 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:33:59.471249 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:33:59.531057 3798194 disk_info.go:125] begin check local disk info of client I1121 20:33:59.534849 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:33:59.534857 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004fe040 0xc0004fe080] E1121 20:34:06.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:34:06.413263 3798194 memory.go:184] no items to output this cycle I1121 20:34:06.413404 3798194 cpu.go:275] no items to output this cycle E1121 20:34:16.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:34:16.413272 3798194 memory.go:184] no items to output this cycle I1121 20:34:16.413421 3798194 cpu.go:275] no items to output this cycle E1121 20:34:26.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:34:26.412887 3798194 memory.go:191] Add success. I1121 20:34:26.417092 3798194 cpu.go:282] Add success. I1121 20:34:26.441619 3798194 net.go:642] Add success. I1121 20:34:26.443053 3798194 net.go:778] primary dev: ETH0 I1121 20:34:26.443068 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:34:26.443082 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:34:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:34:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:34:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:34:36.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:34:36.413266 3798194 memory.go:184] no items to output this cycle I1121 20:34:36.413413 3798194 cpu.go:275] no items to output this cycle E1121 20:34:46.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:34:46.412883 3798194 memory.go:184] no items to output this cycle I1121 20:34:46.413029 3798194 cpu.go:275] no items to output this cycle E1121 20:34:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:34:56.411954 3798194 memory.go:191] Add success. W1121 20:34:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:34:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:34:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:34:56.412342 3798194 cpu.go:282] Add success. I1121 20:34:56.441808 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:34:56.442325 3798194 net.go:642] Add success. I1121 20:34:56.446488 3798194 net.go:778] primary dev: ETH0 I1121 20:34:56.446503 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:34:56.446517 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:34:57.450713 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:34:57.450879 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:34:57.450932 3798194 disk_worker.go:723] disk space is not compliant W1121 20:34:57.450935 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:34:57.459340 3798194 disk_worker.go:509] system disk:vda1 I1121 20:34:57.459383 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:34:58.445948 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:34:59.450427 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:34:59.450511 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:34:59.450537 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:34:59.471071 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:34:59.535184 3798194 disk_info.go:125] begin check local disk info of client I1121 20:34:59.538982 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:34:59.538989 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00039f640 0xc00039f680] E1121 20:35:06.412188 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:35:06.412211 3798194 memory.go:184] no items to output this cycle I1121 20:35:06.412366 3798194 cpu.go:275] no items to output this cycle E1121 20:35:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:35:16.413268 3798194 memory.go:184] no items to output this cycle I1121 20:35:16.413411 3798194 cpu.go:275] no items to output this cycle E1121 20:35:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:35:26.413262 3798194 memory.go:191] Add success. I1121 20:35:26.417763 3798194 cpu.go:282] Add success. I1121 20:35:26.441568 3798194 net.go:642] Add success. I1121 20:35:26.443012 3798194 net.go:778] primary dev: ETH0 I1121 20:35:26.443028 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:35:26.443041 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:35:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:35:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:35:29.450498 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:35:36.412071 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:35:36.412180 3798194 memory.go:184] no items to output this cycle I1121 20:35:36.412607 3798194 cpu.go:275] no items to output this cycle E1121 20:35:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:35:46.413253 3798194 memory.go:184] no items to output this cycle I1121 20:35:46.413406 3798194 cpu.go:275] no items to output this cycle E1121 20:35:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:35:56.411951 3798194 memory.go:191] Add success. W1121 20:35:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:35:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:35:56.412006 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:35:56.412378 3798194 cpu.go:282] Add success. I1121 20:35:56.441598 3798194 event_worker.go:152] Polling the log file for events... I1121 20:35:56.447084 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:35:56.447676 3798194 net.go:642] Add success. I1121 20:35:56.452301 3798194 net.go:778] primary dev: ETH0 I1121 20:35:56.452316 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:35:56.452329 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:35:56.452347 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795262756, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 20:35:56.460163 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 20:35:57.450723 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 20:35:57.450737 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:35:57.450892 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:35:57.450944 3798194 disk_worker.go:723] disk space is not compliant W1121 20:35:57.450947 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:35:57.464223 3798194 disk_worker.go:509] system disk:vda1 I1121 20:35:57.464278 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 20:35:58.448483 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 20:35:58.448498 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 20:35:59.449462 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 20:35:59.450537 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:35:59.450588 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:35:59.450606 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:35:59.471119 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:35:59.539583 3798194 disk_info.go:125] begin check local disk info of client I1121 20:35:59.544251 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:35:59.544262 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00039ef00 0xc00039ef40] E1121 20:36:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:36:06.412261 3798194 memory.go:184] no items to output this cycle I1121 20:36:06.412402 3798194 cpu.go:275] no items to output this cycle E1121 20:36:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:36:16.411974 3798194 memory.go:184] no items to output this cycle I1121 20:36:16.412143 3798194 cpu.go:275] no items to output this cycle I1121 20:36:17.552443 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:36:17.552452 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:36:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:36:26.413265 3798194 memory.go:191] Add success. I1121 20:36:26.417720 3798194 cpu.go:282] Add success. I1121 20:36:26.441657 3798194 net.go:642] Add success. I1121 20:36:26.443068 3798194 net.go:778] primary dev: ETH0 I1121 20:36:26.443082 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:36:26.443095 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:36:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:36:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:36:29.450511 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:36:36.412843 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:36:36.412865 3798194 memory.go:184] no items to output this cycle I1121 20:36:36.413014 3798194 cpu.go:275] no items to output this cycle E1121 20:36:46.412829 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:36:46.412853 3798194 memory.go:184] no items to output this cycle I1121 20:36:46.412996 3798194 cpu.go:275] no items to output this cycle E1121 20:36:56.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:36:56.411959 3798194 memory.go:191] Add success. W1121 20:36:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:36:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:36:56.412002 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:36:56.412348 3798194 cpu.go:282] Add success. I1121 20:36:56.441943 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:36:56.442683 3798194 net.go:642] Add success. I1121 20:36:56.447174 3798194 net.go:778] primary dev: ETH0 I1121 20:36:56.447190 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:36:56.447208 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:36:57.450623 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:36:57.450791 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:36:57.450844 3798194 disk_worker.go:723] disk space is not compliant W1121 20:36:57.450847 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:36:57.459217 3798194 disk_worker.go:509] system disk:vda1 I1121 20:36:57.459260 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:36:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:36:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:36:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:36:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:36:59.471613 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:36:59.545071 3798194 disk_info.go:125] begin check local disk info of client I1121 20:36:59.548853 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:36:59.548860 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071800 0xc000071840] E1121 20:37:06.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:37:06.411987 3798194 memory.go:184] no items to output this cycle I1121 20:37:06.412151 3798194 cpu.go:275] no items to output this cycle E1121 20:37:16.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:37:16.412891 3798194 memory.go:184] no items to output this cycle I1121 20:37:16.413040 3798194 cpu.go:275] no items to output this cycle E1121 20:37:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:37:26.413267 3798194 memory.go:191] Add success. I1121 20:37:26.417552 3798194 cpu.go:282] Add success. I1121 20:37:26.442144 3798194 net.go:642] Add success. I1121 20:37:26.443657 3798194 net.go:778] primary dev: ETH0 I1121 20:37:26.443673 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:37:26.443686 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:37:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:37:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:37:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:37:36.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:37:36.412260 3798194 memory.go:184] no items to output this cycle I1121 20:37:36.412417 3798194 cpu.go:275] no items to output this cycle E1121 20:37:46.411984 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:37:46.412011 3798194 memory.go:184] no items to output this cycle I1121 20:37:46.412182 3798194 cpu.go:275] no items to output this cycle E1121 20:37:56.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:37:56.412887 3798194 memory.go:191] Add success. W1121 20:37:56.412920 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:37:56.412932 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:37:56.412935 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:37:56.416924 3798194 cpu.go:282] Add success. I1121 20:37:56.441353 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:37:56.441918 3798194 net.go:642] Add success. I1121 20:37:56.443413 3798194 net.go:778] primary dev: ETH0 I1121 20:37:56.443429 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:37:56.443441 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:37:57.451228 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:37:57.451398 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:37:57.451455 3798194 disk_worker.go:723] disk space is not compliant W1121 20:37:57.451458 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:37:57.461588 3798194 disk_worker.go:509] system disk:vda1 I1121 20:37:57.461631 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:37:58.445700 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:37:59.449677 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:37:59.449755 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:37:59.449780 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:37:59.471332 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:37:59.549808 3798194 disk_info.go:125] begin check local disk info of client I1121 20:37:59.553603 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:37:59.553611 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004560c0 0xc000456100] E1121 20:38:06.412746 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:38:06.412768 3798194 memory.go:184] no items to output this cycle I1121 20:38:06.412909 3798194 cpu.go:275] no items to output this cycle E1121 20:38:16.412830 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:38:16.412851 3798194 memory.go:184] no items to output this cycle I1121 20:38:16.413170 3798194 cpu.go:275] no items to output this cycle E1121 20:38:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:38:26.411956 3798194 memory.go:191] Add success. I1121 20:38:26.412338 3798194 cpu.go:282] Add success. I1121 20:38:26.441926 3798194 net.go:642] Add success. I1121 20:38:26.445999 3798194 net.go:778] primary dev: ETH0 I1121 20:38:26.446014 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:38:26.446028 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:38:29.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:38:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:38:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:38:36.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:38:36.413268 3798194 memory.go:184] no items to output this cycle I1121 20:38:36.413415 3798194 cpu.go:275] no items to output this cycle E1121 20:38:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:38:46.411974 3798194 memory.go:184] no items to output this cycle I1121 20:38:46.412130 3798194 cpu.go:275] no items to output this cycle E1121 20:38:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:38:56.411945 3798194 memory.go:191] Add success. W1121 20:38:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:38:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:38:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:38:56.412353 3798194 cpu.go:282] Add success. I1121 20:38:56.441840 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:38:56.442352 3798194 net.go:642] Add success. I1121 20:38:56.446622 3798194 net.go:778] primary dev: ETH0 I1121 20:38:56.446636 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:38:56.446651 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:38:57.456192 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:38:57.456361 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:38:57.456415 3798194 disk_worker.go:723] disk space is not compliant W1121 20:38:57.456418 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:38:57.471773 3798194 disk_worker.go:509] system disk:vda1 I1121 20:38:57.471814 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:38:58.445996 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:38:59.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:38:59.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:38:59.450496 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:38:59.471006 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:38:59.554203 3798194 disk_info.go:125] begin check local disk info of client I1121 20:38:59.558151 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:38:59.558161 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000313cc0 0xc000313d00] E1121 20:39:06.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:39:06.412253 3798194 memory.go:184] no items to output this cycle I1121 20:39:06.412392 3798194 cpu.go:275] no items to output this cycle E1121 20:39:16.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:39:16.412260 3798194 memory.go:184] no items to output this cycle I1121 20:39:16.412416 3798194 cpu.go:275] no items to output this cycle I1121 20:39:17.552610 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:39:17.552619 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:39:26.413053 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:39:26.413087 3798194 memory.go:191] Add success. I1121 20:39:26.417707 3798194 cpu.go:282] Add success. I1121 20:39:26.441420 3798194 net.go:642] Add success. I1121 20:39:26.442901 3798194 net.go:778] primary dev: ETH0 I1121 20:39:26.442916 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:39:26.442929 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:39:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:39:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:39:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:39:36.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:39:36.412876 3798194 memory.go:184] no items to output this cycle I1121 20:39:36.413018 3798194 cpu.go:275] no items to output this cycle E1121 20:39:46.412777 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:39:46.412799 3798194 memory.go:184] no items to output this cycle I1121 20:39:46.412943 3798194 cpu.go:275] no items to output this cycle E1121 20:39:56.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:39:56.413277 3798194 memory.go:191] Add success. W1121 20:39:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:39:56.413317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:39:56.413320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:39:56.417581 3798194 cpu.go:282] Add success. I1121 20:39:56.441121 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:39:56.441683 3798194 net.go:642] Add success. I1121 20:39:56.443222 3798194 net.go:778] primary dev: ETH0 I1121 20:39:56.443237 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:39:56.443249 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:39:57.450568 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:39:57.450768 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:39:57.450779 3798194 disk_worker.go:723] disk space is not compliant W1121 20:39:57.450782 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:39:57.459138 3798194 disk_worker.go:509] system disk:vda1 I1121 20:39:57.459178 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:39:58.445688 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:39:59.449909 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:39:59.449978 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:39:59.450003 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:39:59.471530 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:39:59.559057 3798194 disk_info.go:125] begin check local disk info of client I1121 20:39:59.562765 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:39:59.562774 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e1600 0xc0002e1640] E1121 20:40:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:40:06.412257 3798194 memory.go:184] no items to output this cycle I1121 20:40:06.412410 3798194 cpu.go:275] no items to output this cycle E1121 20:40:16.412868 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:40:16.412889 3798194 memory.go:184] no items to output this cycle I1121 20:40:16.413036 3798194 cpu.go:275] no items to output this cycle E1121 20:40:26.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:40:26.412893 3798194 memory.go:191] Add success. I1121 20:40:26.417255 3798194 cpu.go:282] Add success. I1121 20:40:26.441876 3798194 net.go:642] Add success. I1121 20:40:26.443404 3798194 net.go:778] primary dev: ETH0 I1121 20:40:26.443419 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:40:26.443432 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:40:29.450134 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:40:29.450207 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:40:29.450232 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:40:36.413469 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:40:36.413491 3798194 memory.go:184] no items to output this cycle I1121 20:40:36.413644 3798194 cpu.go:275] no items to output this cycle E1121 20:40:46.411649 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:40:46.411671 3798194 memory.go:184] no items to output this cycle I1121 20:40:46.411823 3798194 cpu.go:275] no items to output this cycle E1121 20:40:56.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:40:56.413261 3798194 memory.go:191] Add success. W1121 20:40:56.413292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:40:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:40:56.413307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:40:56.417307 3798194 cpu.go:282] Add success. I1121 20:40:56.444580 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:40:56.445058 3798194 net.go:642] Add success. I1121 20:40:56.446511 3798194 net.go:778] primary dev: ETH0 I1121 20:40:56.446526 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:40:56.446539 3798194 net.go:659] Add eni metrics success, count: 7 E1121 20:40:57.450923 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 20:40:57.450937 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:40:57.451090 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:40:57.451176 3798194 disk_worker.go:723] disk space is not compliant W1121 20:40:57.451179 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:40:57.459868 3798194 disk_worker.go:509] system disk:vda1 I1121 20:40:57.459911 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 20:40:58.448270 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 20:40:58.448285 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 20:40:59.449060 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 20:40:59.450138 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:40:59.450207 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:40:59.450233 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:40:59.470855 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:40:59.563410 3798194 disk_info.go:125] begin check local disk info of client I1121 20:40:59.567162 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:40:59.567171 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00045ea40 0xc00045ea80] E1121 20:41:06.413493 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:41:06.413516 3798194 memory.go:184] no items to output this cycle I1121 20:41:06.413660 3798194 cpu.go:275] no items to output this cycle E1121 20:41:16.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:41:16.413271 3798194 memory.go:184] no items to output this cycle I1121 20:41:16.413416 3798194 cpu.go:275] no items to output this cycle E1121 20:41:26.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:41:26.413268 3798194 memory.go:191] Add success. I1121 20:41:26.417784 3798194 cpu.go:282] Add success. I1121 20:41:26.441243 3798194 net.go:642] Add success. I1121 20:41:26.442943 3798194 net.go:778] primary dev: ETH0 I1121 20:41:26.442958 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:41:26.442971 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:41:29.449643 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:41:29.449723 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:41:29.449752 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:41:36.411980 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:41:36.412003 3798194 memory.go:184] no items to output this cycle I1121 20:41:36.412172 3798194 cpu.go:275] no items to output this cycle E1121 20:41:46.413312 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:41:46.413335 3798194 memory.go:184] no items to output this cycle I1121 20:41:46.413489 3798194 cpu.go:275] no items to output this cycle E1121 20:41:56.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:41:56.412900 3798194 memory.go:191] Add success. W1121 20:41:56.412928 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:41:56.412939 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:41:56.412943 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:41:56.417712 3798194 cpu.go:282] Add success. I1121 20:41:56.441131 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:41:56.441602 3798194 net.go:642] Add success. I1121 20:41:56.443177 3798194 net.go:778] primary dev: ETH0 I1121 20:41:56.443191 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:41:56.443204 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:41:57.450566 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:41:57.450722 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:41:57.450772 3798194 disk_worker.go:723] disk space is not compliant W1121 20:41:57.450776 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:41:57.459245 3798194 disk_worker.go:509] system disk:vda1 I1121 20:41:57.459288 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:41:58.445914 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:41:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:41:59.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:41:59.450504 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:41:59.471023 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:41:59.567256 3798194 disk_info.go:125] begin check local disk info of client I1121 20:41:59.571027 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:41:59.571035 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003bc2c0 0xc0003bc300] E1121 20:42:06.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:42:06.412250 3798194 memory.go:184] no items to output this cycle I1121 20:42:06.412393 3798194 cpu.go:275] no items to output this cycle E1121 20:42:16.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:42:16.413257 3798194 memory.go:184] no items to output this cycle I1121 20:42:16.413403 3798194 cpu.go:275] no items to output this cycle I1121 20:42:17.552781 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:42:17.552789 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:42:26.411415 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:42:26.411448 3798194 memory.go:191] Add success. I1121 20:42:26.415793 3798194 cpu.go:282] Add success. I1121 20:42:26.441416 3798194 net.go:642] Add success. I1121 20:42:26.443105 3798194 net.go:778] primary dev: ETH0 I1121 20:42:26.443134 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:42:26.443154 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:42:29.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:42:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:42:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:42:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:42:36.412254 3798194 memory.go:184] no items to output this cycle I1121 20:42:36.412402 3798194 cpu.go:275] no items to output this cycle E1121 20:42:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:42:46.411984 3798194 memory.go:184] no items to output this cycle I1121 20:42:46.412142 3798194 cpu.go:275] no items to output this cycle E1121 20:42:56.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:42:56.412269 3798194 memory.go:191] Add success. W1121 20:42:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:42:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:42:56.412312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:42:56.417091 3798194 cpu.go:282] Add success. I1121 20:42:56.441549 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:42:56.442034 3798194 net.go:642] Add success. I1121 20:42:56.443577 3798194 net.go:778] primary dev: ETH0 I1121 20:42:56.443593 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:42:56.443605 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:42:57.451086 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:42:57.451270 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:42:57.451323 3798194 disk_worker.go:723] disk space is not compliant W1121 20:42:57.451327 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:42:57.460058 3798194 disk_worker.go:509] system disk:vda1 I1121 20:42:57.460103 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:42:58.446390 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:42:59.450482 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:42:59.450554 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:42:59.450579 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:42:59.471115 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:42:59.571716 3798194 disk_info.go:125] begin check local disk info of client I1121 20:42:59.575502 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:42:59.575512 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00025ed80 0xc00025edc0] E1121 20:43:06.411315 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:43:06.411338 3798194 memory.go:184] no items to output this cycle I1121 20:43:06.411489 3798194 cpu.go:275] no items to output this cycle E1121 20:43:16.412554 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:43:16.412578 3798194 memory.go:184] no items to output this cycle I1121 20:43:16.412727 3798194 cpu.go:275] no items to output this cycle E1121 20:43:26.412287 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:43:26.412323 3798194 memory.go:191] Add success. I1121 20:43:26.416879 3798194 cpu.go:282] Add success. I1121 20:43:26.441304 3798194 net.go:642] Add success. I1121 20:43:26.442830 3798194 net.go:778] primary dev: ETH0 I1121 20:43:26.442845 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:43:26.442858 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:43:29.450309 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:43:29.450387 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:43:29.450414 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:43:36.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:43:36.412878 3798194 memory.go:184] no items to output this cycle I1121 20:43:36.413022 3798194 cpu.go:275] no items to output this cycle E1121 20:43:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:43:46.411973 3798194 memory.go:184] no items to output this cycle I1121 20:43:46.412134 3798194 cpu.go:275] no items to output this cycle E1121 20:43:56.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:43:56.412271 3798194 memory.go:191] Add success. W1121 20:43:56.412298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:43:56.412310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:43:56.412313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:43:56.416528 3798194 cpu.go:282] Add success. I1121 20:43:56.441933 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:43:56.442528 3798194 net.go:642] Add success. I1121 20:43:56.444285 3798194 net.go:778] primary dev: ETH0 I1121 20:43:56.444307 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:43:56.444321 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:43:57.450994 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:43:57.451217 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:43:57.451228 3798194 disk_worker.go:723] disk space is not compliant W1121 20:43:57.451231 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:43:57.460130 3798194 disk_worker.go:509] system disk:vda1 I1121 20:43:57.460171 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:43:58.446337 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:43:59.450407 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:43:59.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:43:59.450504 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:43:59.471040 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:43:59.575612 3798194 disk_info.go:125] begin check local disk info of client I1121 20:43:59.579416 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:43:59.579425 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004d9980 0xc0004d99c0] E1121 20:44:06.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:44:06.411992 3798194 memory.go:184] no items to output this cycle I1121 20:44:06.412149 3798194 cpu.go:275] no items to output this cycle E1121 20:44:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:44:16.411973 3798194 memory.go:184] no items to output this cycle I1121 20:44:16.412125 3798194 cpu.go:275] no items to output this cycle E1121 20:44:26.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:44:26.412893 3798194 memory.go:191] Add success. I1121 20:44:26.417004 3798194 cpu.go:282] Add success. I1121 20:44:26.441551 3798194 net.go:642] Add success. I1121 20:44:26.443032 3798194 net.go:778] primary dev: ETH0 I1121 20:44:26.443048 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:44:26.443061 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:44:29.449617 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:44:29.449698 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:44:29.449723 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:44:36.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:44:36.412880 3798194 memory.go:184] no items to output this cycle I1121 20:44:36.413023 3798194 cpu.go:275] no items to output this cycle E1121 20:44:46.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:44:46.412250 3798194 memory.go:184] no items to output this cycle I1121 20:44:46.412393 3798194 cpu.go:275] no items to output this cycle E1121 20:44:56.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:44:56.413265 3798194 memory.go:191] Add success. W1121 20:44:56.413290 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:44:56.413302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:44:56.413305 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:44:56.417241 3798194 cpu.go:282] Add success. I1121 20:44:56.441628 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:44:56.442102 3798194 net.go:642] Add success. I1121 20:44:56.443544 3798194 net.go:778] primary dev: ETH0 I1121 20:44:56.443561 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:44:56.443573 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:44:57.451000 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:44:57.451221 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:44:57.451233 3798194 disk_worker.go:723] disk space is not compliant W1121 20:44:57.451236 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:44:57.460106 3798194 disk_worker.go:509] system disk:vda1 I1121 20:44:57.460170 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:44:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:44:59.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:44:59.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:44:59.450503 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:44:59.471024 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:44:59.579511 3798194 disk_info.go:125] begin check local disk info of client I1121 20:44:59.583189 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:44:59.583199 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000436040 0xc000436080] E1121 20:45:06.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:45:06.413257 3798194 memory.go:184] no items to output this cycle I1121 20:45:06.413398 3798194 cpu.go:275] no items to output this cycle E1121 20:45:16.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:45:16.412879 3798194 memory.go:184] no items to output this cycle I1121 20:45:16.413026 3798194 cpu.go:275] no items to output this cycle I1121 20:45:17.553224 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:45:17.553233 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:45:26.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:45:26.413280 3798194 memory.go:191] Add success. I1121 20:45:26.417408 3798194 cpu.go:282] Add success. I1121 20:45:26.441834 3798194 net.go:642] Add success. I1121 20:45:26.443225 3798194 net.go:778] primary dev: ETH0 I1121 20:45:26.443239 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:45:26.443253 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:45:29.450456 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:45:29.450543 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:45:29.450585 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:45:36.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:45:36.411976 3798194 memory.go:184] no items to output this cycle I1121 20:45:36.412137 3798194 cpu.go:275] no items to output this cycle E1121 20:45:46.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:45:46.412884 3798194 memory.go:184] no items to output this cycle I1121 20:45:46.413030 3798194 cpu.go:275] no items to output this cycle E1121 20:45:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:45:56.413267 3798194 memory.go:191] Add success. W1121 20:45:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:45:56.413309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:45:56.413312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:45:56.427212 3798194 cpu.go:282] Add success. I1121 20:45:56.441472 3798194 event_worker.go:152] Polling the log file for events... I1121 20:45:56.444621 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:45:56.445106 3798194 net.go:642] Add success. I1121 20:45:56.446560 3798194 net.go:778] primary dev: ETH0 I1121 20:45:56.446574 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:45:56.446587 3798194 net.go:659] Add eni metrics success, count: 7 E1121 20:45:57.450735 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 20:45:57.450748 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:45:57.450917 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:45:57.450967 3798194 disk_worker.go:723] disk space is not compliant W1121 20:45:57.450970 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:45:57.459370 3798194 disk_worker.go:509] system disk:vda1 I1121 20:45:57.459412 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 20:45:58.448013 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 20:45:58.448027 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:45:59.451687 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:45:59.451774 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:45:59.451808 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:45:59.452877 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 20:45:59.474158 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:45:59.584168 3798194 disk_info.go:125] begin check local disk info of client I1121 20:45:59.589664 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:45:59.589677 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000323580 0xc0003235c0] E1121 20:46:06.412767 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:46:06.412794 3798194 memory.go:184] no items to output this cycle I1121 20:46:06.412938 3798194 cpu.go:275] no items to output this cycle E1121 20:46:16.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:46:16.412887 3798194 memory.go:184] no items to output this cycle I1121 20:46:16.413031 3798194 cpu.go:275] no items to output this cycle E1121 20:46:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:46:26.411946 3798194 memory.go:191] Add success. I1121 20:46:26.412311 3798194 cpu.go:282] Add success. I1121 20:46:26.442171 3798194 net.go:642] Add success. I1121 20:46:26.446193 3798194 net.go:778] primary dev: ETH0 I1121 20:46:26.446208 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:46:26.446222 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:46:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:46:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:46:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:46:36.413266 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:46:36.413289 3798194 memory.go:184] no items to output this cycle I1121 20:46:36.413430 3798194 cpu.go:275] no items to output this cycle E1121 20:46:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:46:46.413259 3798194 memory.go:184] no items to output this cycle I1121 20:46:46.413404 3798194 cpu.go:275] no items to output this cycle E1121 20:46:56.412419 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:46:56.412467 3798194 memory.go:191] Add success. W1121 20:46:56.412498 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:46:56.412510 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:46:56.412515 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:46:56.417412 3798194 cpu.go:282] Add success. I1121 20:46:56.440944 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:46:56.441534 3798194 net.go:642] Add success. I1121 20:46:56.442966 3798194 net.go:778] primary dev: ETH0 I1121 20:46:56.442980 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:46:56.442992 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:46:57.450521 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:46:57.450720 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:46:57.450732 3798194 disk_worker.go:723] disk space is not compliant W1121 20:46:57.450735 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:46:57.459128 3798194 disk_worker.go:509] system disk:vda1 I1121 20:46:57.459171 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:46:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:46:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:46:59.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:46:59.450150 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:46:59.471683 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:46:59.590375 3798194 disk_info.go:125] begin check local disk info of client I1121 20:46:59.594197 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:46:59.594206 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ee00 0xc00030ee40] E1121 20:47:06.412001 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:47:06.412023 3798194 memory.go:184] no items to output this cycle I1121 20:47:06.412196 3798194 cpu.go:275] no items to output this cycle E1121 20:47:16.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:47:16.413258 3798194 memory.go:184] no items to output this cycle I1121 20:47:16.413403 3798194 cpu.go:275] no items to output this cycle E1121 20:47:26.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:47:26.412898 3798194 memory.go:191] Add success. I1121 20:47:26.417030 3798194 cpu.go:282] Add success. I1121 20:47:26.441691 3798194 net.go:642] Add success. I1121 20:47:26.443065 3798194 net.go:778] primary dev: ETH0 I1121 20:47:26.443080 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:47:26.443092 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:47:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:47:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:47:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:47:36.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:47:36.412254 3798194 memory.go:184] no items to output this cycle I1121 20:47:36.412396 3798194 cpu.go:275] no items to output this cycle E1121 20:47:46.412559 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:47:46.412581 3798194 memory.go:184] no items to output this cycle I1121 20:47:46.412726 3798194 cpu.go:275] no items to output this cycle E1121 20:47:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:47:56.411950 3798194 memory.go:191] Add success. W1121 20:47:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:47:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:47:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:47:56.414273 3798194 cpu.go:282] Add success. I1121 20:47:56.441226 3798194 net.go:642] Add success. I1121 20:47:56.451794 3798194 net.go:778] primary dev: ETH0 I1121 20:47:56.451810 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:47:56.451825 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:47:56.441563 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:47:57.445470 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:47:57.445672 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:47:57.445746 3798194 disk_worker.go:723] disk space is not compliant W1121 20:47:57.445749 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:47:57.447846 3798194 disk_worker.go:509] system disk:vda1 I1121 20:47:57.447885 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:47:58.445987 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:47:59.449971 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:47:59.450056 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:47:59.450087 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:47:59.471576 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:47:59.595197 3798194 disk_info.go:125] begin check local disk info of client I1121 20:47:59.599076 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:47:59.599085 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00048a140 0xc00048a180] E1121 20:48:06.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:48:06.412366 3798194 memory.go:184] no items to output this cycle I1121 20:48:06.412798 3798194 cpu.go:275] no items to output this cycle E1121 20:48:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:48:16.411979 3798194 memory.go:184] no items to output this cycle I1121 20:48:16.412144 3798194 cpu.go:275] no items to output this cycle I1121 20:48:17.553395 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:48:17.553404 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:48:26.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:48:26.412275 3798194 memory.go:191] Add success. I1121 20:48:26.416870 3798194 cpu.go:282] Add success. I1121 20:48:26.441558 3798194 net.go:642] Add success. I1121 20:48:26.443663 3798194 net.go:778] primary dev: ETH0 I1121 20:48:26.443682 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:48:26.443696 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:48:29.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:48:29.450467 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:48:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:48:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:48:36.412259 3798194 memory.go:184] no items to output this cycle I1121 20:48:36.412403 3798194 cpu.go:275] no items to output this cycle E1121 20:48:46.412221 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:48:46.412243 3798194 memory.go:184] no items to output this cycle I1121 20:48:46.412395 3798194 cpu.go:275] no items to output this cycle E1121 20:48:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:48:56.411954 3798194 memory.go:191] Add success. W1121 20:48:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:48:56.412005 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:48:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:48:56.412358 3798194 cpu.go:282] Add success. I1121 20:48:56.441929 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:48:56.442526 3798194 net.go:642] Add success. I1121 20:48:56.446637 3798194 net.go:778] primary dev: ETH0 I1121 20:48:56.446651 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:48:56.446665 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:48:57.450665 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:48:57.450830 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:48:57.450886 3798194 disk_worker.go:723] disk space is not compliant W1121 20:48:57.450889 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:48:57.459651 3798194 disk_worker.go:509] system disk:vda1 I1121 20:48:57.459694 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:48:58.446377 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:48:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:48:59.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:48:59.450148 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:48:59.470679 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:48:59.599187 3798194 disk_info.go:125] begin check local disk info of client I1121 20:48:59.603335 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:48:59.603344 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037d340 0xc00037d380] E1121 20:49:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:49:06.412258 3798194 memory.go:184] no items to output this cycle I1121 20:49:06.412411 3798194 cpu.go:275] no items to output this cycle E1121 20:49:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:49:16.411989 3798194 memory.go:184] no items to output this cycle I1121 20:49:16.412145 3798194 cpu.go:275] no items to output this cycle E1121 20:49:26.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:49:26.413280 3798194 memory.go:191] Add success. I1121 20:49:26.417461 3798194 cpu.go:282] Add success. I1121 20:49:26.441972 3798194 net.go:642] Add success. I1121 20:49:26.443515 3798194 net.go:778] primary dev: ETH0 I1121 20:49:26.443530 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:49:26.443542 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:49:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:49:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:49:29.450495 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:49:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:49:36.413253 3798194 memory.go:184] no items to output this cycle I1121 20:49:36.413688 3798194 cpu.go:275] no items to output this cycle E1121 20:49:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:49:46.412258 3798194 memory.go:184] no items to output this cycle I1121 20:49:46.412406 3798194 cpu.go:275] no items to output this cycle E1121 20:49:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:49:56.411955 3798194 memory.go:191] Add success. W1121 20:49:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:49:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:49:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:49:56.412361 3798194 cpu.go:282] Add success. I1121 20:49:56.441803 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:49:56.442299 3798194 net.go:642] Add success. I1121 20:49:56.446201 3798194 net.go:778] primary dev: ETH0 I1121 20:49:56.446218 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:49:56.446231 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:49:57.452854 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:49:57.453020 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:49:57.453075 3798194 disk_worker.go:723] disk space is not compliant W1121 20:49:57.453078 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:49:57.465481 3798194 disk_worker.go:509] system disk:vda1 I1121 20:49:57.465525 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:49:58.445986 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:49:59.450500 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:49:59.450576 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:49:59.450600 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:49:59.471092 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:49:59.603443 3798194 disk_info.go:125] begin check local disk info of client I1121 20:49:59.607371 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:49:59.607381 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b8e40 0xc0003b8e80] E1121 20:50:06.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:50:06.412261 3798194 memory.go:184] no items to output this cycle I1121 20:50:06.412409 3798194 cpu.go:275] no items to output this cycle E1121 20:50:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:50:16.411978 3798194 memory.go:184] no items to output this cycle I1121 20:50:16.412135 3798194 cpu.go:275] no items to output this cycle E1121 20:50:26.412888 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:50:26.412921 3798194 memory.go:191] Add success. I1121 20:50:26.417380 3798194 cpu.go:282] Add success. I1121 20:50:26.442177 3798194 net.go:642] Add success. I1121 20:50:26.443616 3798194 net.go:778] primary dev: ETH0 I1121 20:50:26.443637 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:50:26.443652 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:50:29.450007 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:50:29.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:50:29.450124 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:50:36.412834 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:50:36.412861 3798194 memory.go:184] no items to output this cycle I1121 20:50:36.413006 3798194 cpu.go:275] no items to output this cycle E1121 20:50:46.411104 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:50:46.411148 3798194 memory.go:184] no items to output this cycle I1121 20:50:46.411320 3798194 cpu.go:275] no items to output this cycle E1121 20:50:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:50:56.413264 3798194 memory.go:191] Add success. W1121 20:50:56.413295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:50:56.413309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:50:56.413312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:50:56.417536 3798194 cpu.go:282] Add success. I1121 20:50:56.446795 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:50:56.447315 3798194 net.go:642] Add success. I1121 20:50:56.448952 3798194 net.go:778] primary dev: ETH0 I1121 20:50:56.449136 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:50:56.449153 3798194 net.go:659] Add eni metrics success, count: 7 E1121 20:50:57.452636 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 20:50:57.452649 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:50:57.452860 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:50:57.452869 3798194 disk_worker.go:723] disk space is not compliant W1121 20:50:57.452873 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:50:57.462277 3798194 disk_worker.go:509] system disk:vda1 I1121 20:50:57.462317 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 20:50:58.448451 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 20:50:58.448465 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 20:50:59.449193 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 20:50:59.450267 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:50:59.450319 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:50:59.450343 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:50:59.470874 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:50:59.607479 3798194 disk_info.go:125] begin check local disk info of client I1121 20:50:59.611301 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:50:59.611310 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000070f40 0xc000070f80] E1121 20:51:06.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:51:06.413271 3798194 memory.go:184] no items to output this cycle I1121 20:51:06.413415 3798194 cpu.go:275] no items to output this cycle E1121 20:51:16.412796 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:51:16.412818 3798194 memory.go:184] no items to output this cycle I1121 20:51:16.412962 3798194 cpu.go:275] no items to output this cycle I1121 20:51:17.553572 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:51:17.553583 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:51:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:51:26.413268 3798194 memory.go:191] Add success. I1121 20:51:26.417760 3798194 cpu.go:282] Add success. I1121 20:51:26.442275 3798194 net.go:642] Add success. I1121 20:51:26.443755 3798194 net.go:778] primary dev: ETH0 I1121 20:51:26.443769 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:51:26.443783 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:51:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:51:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:51:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:51:36.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:51:36.413250 3798194 memory.go:184] no items to output this cycle I1121 20:51:36.413392 3798194 cpu.go:275] no items to output this cycle E1121 20:51:46.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:51:46.412255 3798194 memory.go:184] no items to output this cycle I1121 20:51:46.412402 3798194 cpu.go:275] no items to output this cycle E1121 20:51:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:51:56.413267 3798194 memory.go:191] Add success. W1121 20:51:56.413302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:51:56.413314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:51:56.413318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:51:56.417722 3798194 cpu.go:282] Add success. I1121 20:51:56.441096 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:51:56.441683 3798194 net.go:642] Add success. I1121 20:51:56.443213 3798194 net.go:778] primary dev: ETH0 I1121 20:51:56.443228 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:51:56.443239 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:51:57.450849 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:51:57.451048 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:51:57.451058 3798194 disk_worker.go:723] disk space is not compliant W1121 20:51:57.451061 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:51:57.459782 3798194 disk_worker.go:509] system disk:vda1 I1121 20:51:57.459824 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:51:58.446370 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:51:59.449676 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:51:59.449762 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:51:59.449788 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:51:59.471323 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:51:59.612130 3798194 disk_info.go:125] begin check local disk info of client I1121 20:51:59.615950 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:51:59.615959 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ef6c0 0xc0003ef700] E1121 20:52:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:52:06.412253 3798194 memory.go:184] no items to output this cycle I1121 20:52:06.412395 3798194 cpu.go:275] no items to output this cycle E1121 20:52:16.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:52:16.413264 3798194 memory.go:184] no items to output this cycle I1121 20:52:16.413617 3798194 cpu.go:275] no items to output this cycle E1121 20:52:26.411234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:52:26.411268 3798194 memory.go:191] Add success. I1121 20:52:26.416086 3798194 cpu.go:282] Add success. I1121 20:52:26.441688 3798194 net.go:642] Add success. I1121 20:52:26.443275 3798194 net.go:778] primary dev: ETH0 I1121 20:52:26.443290 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:52:26.443303 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:52:29.450055 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:52:29.450147 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:52:29.450176 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:52:36.413137 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:52:36.413172 3798194 memory.go:184] no items to output this cycle I1121 20:52:36.413323 3798194 cpu.go:275] no items to output this cycle E1121 20:52:46.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:52:46.412261 3798194 memory.go:184] no items to output this cycle I1121 20:52:46.412417 3798194 cpu.go:275] no items to output this cycle E1121 20:52:56.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:52:56.412883 3798194 memory.go:191] Add success. W1121 20:52:56.412916 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:52:56.412928 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:52:56.412931 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:52:56.417286 3798194 cpu.go:282] Add success. I1121 20:52:56.441680 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:52:56.442202 3798194 net.go:642] Add success. I1121 20:52:56.443695 3798194 net.go:778] primary dev: ETH0 I1121 20:52:56.443717 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:52:56.443732 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:52:57.453386 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:52:57.453595 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:52:57.453606 3798194 disk_worker.go:723] disk space is not compliant W1121 20:52:57.453610 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:52:57.463641 3798194 disk_worker.go:509] system disk:vda1 I1121 20:52:57.463696 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:52:58.445993 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:52:59.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:52:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:52:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:52:59.471659 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:52:59.616063 3798194 disk_info.go:125] begin check local disk info of client I1121 20:52:59.619953 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:52:59.619963 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002ffcc0 0xc0002ffd00] E1121 20:53:06.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:53:06.412880 3798194 memory.go:184] no items to output this cycle I1121 20:53:06.413017 3798194 cpu.go:275] no items to output this cycle E1121 20:53:16.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:53:16.412251 3798194 memory.go:184] no items to output this cycle I1121 20:53:16.412399 3798194 cpu.go:275] no items to output this cycle E1121 20:53:26.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:53:26.412269 3798194 memory.go:191] Add success. I1121 20:53:26.416507 3798194 cpu.go:282] Add success. I1121 20:53:26.442067 3798194 net.go:642] Add success. I1121 20:53:26.443466 3798194 net.go:778] primary dev: ETH0 I1121 20:53:26.443479 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:53:26.443493 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:53:29.450403 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:53:29.450493 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:53:29.450519 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:53:36.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:53:36.412872 3798194 memory.go:184] no items to output this cycle I1121 20:53:36.413022 3798194 cpu.go:275] no items to output this cycle E1121 20:53:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:53:46.411981 3798194 memory.go:184] no items to output this cycle I1121 20:53:46.412139 3798194 cpu.go:275] no items to output this cycle E1121 20:53:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:53:56.411943 3798194 memory.go:191] Add success. W1121 20:53:56.411970 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:53:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:53:56.411985 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:53:56.412370 3798194 cpu.go:282] Add success. I1121 20:53:56.441822 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:53:56.442331 3798194 net.go:642] Add success. I1121 20:53:56.446809 3798194 net.go:778] primary dev: ETH0 I1121 20:53:56.446824 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:53:56.446837 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:53:57.451051 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:53:57.451275 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:53:57.451286 3798194 disk_worker.go:723] disk space is not compliant W1121 20:53:57.451290 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:53:57.460020 3798194 disk_worker.go:509] system disk:vda1 I1121 20:53:57.460060 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:53:58.445993 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:53:59.450091 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:53:59.450174 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:53:59.450205 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:53:59.470685 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:53:59.620061 3798194 disk_info.go:125] begin check local disk info of client I1121 20:53:59.623910 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:53:59.623919 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000381f00 0xc000381f40] E1121 20:54:06.412987 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:54:06.413012 3798194 memory.go:184] no items to output this cycle I1121 20:54:06.413178 3798194 cpu.go:275] no items to output this cycle E1121 20:54:16.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:54:16.411989 3798194 memory.go:184] no items to output this cycle I1121 20:54:16.412151 3798194 cpu.go:275] no items to output this cycle I1121 20:54:17.554627 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:54:17.554637 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:54:26.411537 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:54:26.411570 3798194 memory.go:191] Add success. I1121 20:54:26.411910 3798194 cpu.go:282] Add success. I1121 20:54:26.441550 3798194 net.go:642] Add success. I1121 20:54:26.445647 3798194 net.go:778] primary dev: ETH0 I1121 20:54:26.445661 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:54:26.445676 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:54:29.449903 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:54:29.449981 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:54:29.450007 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:54:36.412826 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:54:36.412848 3798194 memory.go:184] no items to output this cycle I1121 20:54:36.412999 3798194 cpu.go:275] no items to output this cycle E1121 20:54:46.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:54:46.413266 3798194 memory.go:184] no items to output this cycle I1121 20:54:46.413412 3798194 cpu.go:275] no items to output this cycle E1121 20:54:56.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:54:56.412264 3798194 memory.go:191] Add success. W1121 20:54:56.412298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:54:56.412310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:54:56.412313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:54:56.416511 3798194 cpu.go:282] Add success. I1121 20:54:56.441079 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:54:56.441590 3798194 net.go:642] Add success. I1121 20:54:56.443030 3798194 net.go:778] primary dev: ETH0 I1121 20:54:56.443044 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:54:56.443057 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:54:57.452656 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:54:57.452820 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:54:57.452874 3798194 disk_worker.go:723] disk space is not compliant W1121 20:54:57.452877 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:54:57.463369 3798194 disk_worker.go:509] system disk:vda1 I1121 20:54:57.463411 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:54:58.446021 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:54:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:54:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:54:59.450130 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:54:59.471664 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:54:59.624014 3798194 disk_info.go:125] begin check local disk info of client I1121 20:54:59.627933 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:54:59.627942 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000487600 0xc000487640] E1121 20:55:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:55:06.411978 3798194 memory.go:184] no items to output this cycle I1121 20:55:06.412144 3798194 cpu.go:275] no items to output this cycle E1121 20:55:16.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:55:16.413254 3798194 memory.go:184] no items to output this cycle I1121 20:55:16.413403 3798194 cpu.go:275] no items to output this cycle E1121 20:55:26.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:55:26.412261 3798194 memory.go:191] Add success. I1121 20:55:26.416325 3798194 cpu.go:282] Add success. I1121 20:55:26.441978 3798194 net.go:642] Add success. I1121 20:55:26.443332 3798194 net.go:778] primary dev: ETH0 I1121 20:55:26.443345 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:55:26.443358 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:55:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:55:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:55:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:55:36.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:55:36.412270 3798194 memory.go:184] no items to output this cycle I1121 20:55:36.412411 3798194 cpu.go:275] no items to output this cycle E1121 20:55:46.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:55:46.412267 3798194 memory.go:184] no items to output this cycle I1121 20:55:46.412417 3798194 cpu.go:275] no items to output this cycle E1121 20:55:56.412972 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:55:56.413003 3798194 memory.go:191] Add success. W1121 20:55:56.413039 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:55:56.413051 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:55:56.413054 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:55:56.417681 3798194 cpu.go:282] Add success. I1121 20:55:56.443843 3798194 event_worker.go:152] Polling the log file for events... I1121 20:55:56.444064 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:55:56.444558 3798194 net.go:642] Add success. I1121 20:55:56.446099 3798194 net.go:778] primary dev: ETH0 I1121 20:55:56.446126 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:55:56.446139 3798194 net.go:659] Add eni metrics success, count: 7 E1121 20:55:57.450543 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 20:55:57.450556 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:55:57.450707 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:55:57.450757 3798194 disk_worker.go:723] disk space is not compliant W1121 20:55:57.450760 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:55:57.459045 3798194 disk_worker.go:509] system disk:vda1 I1121 20:55:57.459087 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 20:55:58.448672 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 20:55:58.448687 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:55:59.449951 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:55:59.450015 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:55:59.450041 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:55:59.450216 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 20:55:59.470708 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:55:59.628036 3798194 disk_info.go:125] begin check local disk info of client I1121 20:55:59.631679 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:55:59.631687 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003cad00 0xc0003cad40] E1121 20:56:06.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:56:06.411978 3798194 memory.go:184] no items to output this cycle I1121 20:56:06.412132 3798194 cpu.go:275] no items to output this cycle E1121 20:56:16.412327 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:56:16.412350 3798194 memory.go:184] no items to output this cycle I1121 20:56:16.412493 3798194 cpu.go:275] no items to output this cycle E1121 20:56:26.412707 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:56:26.412740 3798194 memory.go:191] Add success. I1121 20:56:26.417288 3798194 cpu.go:282] Add success. I1121 20:56:26.441500 3798194 net.go:642] Add success. I1121 20:56:26.443011 3798194 net.go:778] primary dev: ETH0 I1121 20:56:26.443026 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:56:26.443040 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:56:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:56:29.450131 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:56:29.450160 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:56:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:56:36.411978 3798194 memory.go:184] no items to output this cycle I1121 20:56:36.412132 3798194 cpu.go:275] no items to output this cycle E1121 20:56:46.412779 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:56:46.412801 3798194 memory.go:184] no items to output this cycle I1121 20:56:46.412947 3798194 cpu.go:275] no items to output this cycle E1121 20:56:56.412541 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:56:56.412571 3798194 memory.go:191] Add success. W1121 20:56:56.412604 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:56:56.412615 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:56:56.412618 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:56:56.416942 3798194 cpu.go:282] Add success. I1121 20:56:56.441326 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:56:56.441899 3798194 net.go:642] Add success. I1121 20:56:56.443393 3798194 net.go:778] primary dev: ETH0 I1121 20:56:56.443407 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:56:56.443419 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:56:57.453096 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:56:57.453330 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:56:57.453342 3798194 disk_worker.go:723] disk space is not compliant W1121 20:56:57.453345 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:56:57.461714 3798194 disk_worker.go:509] system disk:vda1 I1121 20:56:57.461756 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:56:58.446372 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:56:59.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:56:59.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:56:59.450495 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:56:59.471007 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:56:59.631782 3798194 disk_info.go:125] begin check local disk info of client I1121 20:56:59.635612 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:56:59.635620 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00043b140 0xc00043b180] E1121 20:57:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:57:06.412260 3798194 memory.go:184] no items to output this cycle I1121 20:57:06.412402 3798194 cpu.go:275] no items to output this cycle E1121 20:57:16.413221 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:57:16.413243 3798194 memory.go:184] no items to output this cycle I1121 20:57:16.413396 3798194 cpu.go:275] no items to output this cycle I1121 20:57:17.554798 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 20:57:17.554807 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 20:57:26.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:57:26.412264 3798194 memory.go:191] Add success. I1121 20:57:26.416605 3798194 cpu.go:282] Add success. I1121 20:57:26.442133 3798194 net.go:642] Add success. I1121 20:57:26.443733 3798194 net.go:778] primary dev: ETH0 I1121 20:57:26.443748 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:57:26.443765 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:57:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:57:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:57:29.450129 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:57:36.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:57:36.411967 3798194 memory.go:184] no items to output this cycle I1121 20:57:36.412133 3798194 cpu.go:275] no items to output this cycle E1121 20:57:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:57:46.413254 3798194 memory.go:184] no items to output this cycle I1121 20:57:46.413396 3798194 cpu.go:275] no items to output this cycle E1121 20:57:56.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:57:56.412259 3798194 memory.go:191] Add success. W1121 20:57:56.412290 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:57:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:57:56.412305 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:57:56.416668 3798194 cpu.go:282] Add success. I1121 20:57:56.440949 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:57:56.441529 3798194 net.go:642] Add success. I1121 20:57:56.443034 3798194 net.go:778] primary dev: ETH0 I1121 20:57:56.443049 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:57:56.443063 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:57:57.450979 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:57:57.451213 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:57:57.451224 3798194 disk_worker.go:723] disk space is not compliant W1121 20:57:57.451228 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:57:57.461600 3798194 disk_worker.go:509] system disk:vda1 I1121 20:57:57.461642 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:57:58.446012 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:57:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:57:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:57:59.450154 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:57:59.470671 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:57:59.636651 3798194 disk_info.go:125] begin check local disk info of client I1121 20:57:59.641286 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:57:59.641297 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e0c0 0xc00030e100] E1121 20:58:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:58:06.412254 3798194 memory.go:184] no items to output this cycle I1121 20:58:06.412395 3798194 cpu.go:275] no items to output this cycle E1121 20:58:16.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:58:16.413249 3798194 memory.go:184] no items to output this cycle I1121 20:58:16.413398 3798194 cpu.go:275] no items to output this cycle E1121 20:58:26.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:58:26.412256 3798194 memory.go:191] Add success. I1121 20:58:26.416520 3798194 cpu.go:282] Add success. I1121 20:58:26.442183 3798194 net.go:642] Add success. I1121 20:58:26.443727 3798194 net.go:778] primary dev: ETH0 I1121 20:58:26.443743 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:58:26.443755 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:58:29.450521 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:58:29.450595 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:58:29.450619 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:58:36.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:58:36.412249 3798194 memory.go:184] no items to output this cycle I1121 20:58:36.412388 3798194 cpu.go:275] no items to output this cycle E1121 20:58:46.412257 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:58:46.412285 3798194 memory.go:184] no items to output this cycle I1121 20:58:46.412431 3798194 cpu.go:275] no items to output this cycle E1121 20:58:56.411930 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:58:56.411974 3798194 memory.go:191] Add success. W1121 20:58:56.412017 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:58:56.412033 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:58:56.412038 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:58:56.412502 3798194 cpu.go:282] Add success. I1121 20:58:56.441616 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:58:56.442099 3798194 net.go:642] Add success. I1121 20:58:56.446244 3798194 net.go:778] primary dev: ETH0 I1121 20:58:56.446259 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:58:56.446273 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:58:57.451090 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:58:57.451327 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:58:57.451339 3798194 disk_worker.go:723] disk space is not compliant W1121 20:58:57.451342 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:58:57.459737 3798194 disk_worker.go:509] system disk:vda1 I1121 20:58:57.459782 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:58:58.446018 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:58:59.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:58:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:58:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:58:59.471650 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:58:59.641390 3798194 disk_info.go:125] begin check local disk info of client I1121 20:58:59.645409 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:58:59.645418 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fdbc0 0xc0003fdc00] E1121 20:59:06.412355 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:59:06.412445 3798194 memory.go:184] no items to output this cycle I1121 20:59:06.412781 3798194 cpu.go:275] no items to output this cycle E1121 20:59:16.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:59:16.413247 3798194 memory.go:184] no items to output this cycle I1121 20:59:16.413408 3798194 cpu.go:275] no items to output this cycle E1121 20:59:26.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:59:26.413275 3798194 memory.go:191] Add success. I1121 20:59:26.417516 3798194 cpu.go:282] Add success. I1121 20:59:26.442168 3798194 net.go:642] Add success. I1121 20:59:26.443495 3798194 net.go:778] primary dev: ETH0 I1121 20:59:26.443510 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:59:26.443524 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:59:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:59:29.450465 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:59:29.450493 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 20:59:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:59:36.412265 3798194 memory.go:184] no items to output this cycle I1121 20:59:36.412409 3798194 cpu.go:275] no items to output this cycle E1121 20:59:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:59:46.411979 3798194 memory.go:184] no items to output this cycle I1121 20:59:46.412141 3798194 cpu.go:275] no items to output this cycle E1121 20:59:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 20:59:56.411948 3798194 memory.go:191] Add success. W1121 20:59:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 20:59:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 20:59:56.411991 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 20:59:56.412354 3798194 cpu.go:282] Add success. I1121 20:59:56.441803 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 20:59:56.442326 3798194 net.go:642] Add success. I1121 20:59:56.446427 3798194 net.go:778] primary dev: ETH0 I1121 20:59:56.446443 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 20:59:56.446455 3798194 net.go:659] Add eni metrics success, count: 7 I1121 20:59:57.450600 3798194 custom_config.go:106] query custom config with name: gpu W1121 20:59:57.450796 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 20:59:57.450807 3798194 disk_worker.go:723] disk space is not compliant W1121 20:59:57.450810 3798194 disk_worker.go:743] disk inode is not compliant I1121 20:59:57.459094 3798194 disk_worker.go:509] system disk:vda1 I1121 20:59:57.459148 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 20:59:58.445992 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 20:59:59.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 20:59:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 20:59:59.450130 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 20:59:59.471651 3798194 disk_local_worker.go:436] Get disk info: [] I1121 20:59:59.645516 3798194 disk_info.go:125] begin check local disk info of client I1121 20:59:59.649360 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 20:59:59.649369 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036d340 0xc00036d380] E1121 21:00:06.412761 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:00:06.412785 3798194 memory.go:184] no items to output this cycle I1121 21:00:06.412943 3798194 cpu.go:275] no items to output this cycle E1121 21:00:16.411975 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:00:16.412000 3798194 memory.go:184] no items to output this cycle I1121 21:00:16.412159 3798194 cpu.go:275] no items to output this cycle I1121 21:00:17.554967 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:00:17.554976 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:00:26.411320 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:00:26.411352 3798194 memory.go:191] Add success. I1121 21:00:26.411693 3798194 cpu.go:282] Add success. I1121 21:00:26.441574 3798194 net.go:642] Add success. I1121 21:00:26.446690 3798194 net.go:778] primary dev: ETH0 I1121 21:00:26.446705 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:00:26.446719 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:00:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:00:29.450140 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:00:29.450171 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:00:36.413156 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:00:36.413180 3798194 memory.go:184] no items to output this cycle I1121 21:00:36.413324 3798194 cpu.go:275] no items to output this cycle E1121 21:00:46.413407 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:00:46.413431 3798194 memory.go:184] no items to output this cycle I1121 21:00:46.413577 3798194 cpu.go:275] no items to output this cycle E1121 21:00:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:00:56.411945 3798194 memory.go:191] Add success. W1121 21:00:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:00:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:00:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:00:56.412328 3798194 cpu.go:282] Add success. I1121 21:00:56.445914 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:00:56.446463 3798194 net.go:642] Add success. I1121 21:00:56.452062 3798194 net.go:778] primary dev: ETH0 I1121 21:00:56.452078 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:00:56.452090 3798194 net.go:659] Add eni metrics success, count: 7 E1121 21:00:57.451094 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 21:00:57.451124 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:00:57.451330 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:00:57.451339 3798194 disk_worker.go:723] disk space is not compliant W1121 21:00:57.451342 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:00:57.459799 3798194 disk_worker.go:509] system disk:vda1 I1121 21:00:57.459841 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 21:00:58.447991 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 21:00:58.448006 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 21:00:59.448642 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 21:00:59.449719 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:00:59.449771 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:00:59.449795 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:00:59.471335 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:00:59.650140 3798194 disk_info.go:125] begin check local disk info of client I1121 21:00:59.653908 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:00:59.653916 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00025c140 0xc00025c180] E1121 21:01:06.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:01:06.412874 3798194 memory.go:184] no items to output this cycle I1121 21:01:06.413025 3798194 cpu.go:275] no items to output this cycle E1121 21:01:16.411417 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:01:16.411438 3798194 memory.go:184] no items to output this cycle I1121 21:01:16.411582 3798194 cpu.go:275] no items to output this cycle E1121 21:01:26.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:01:26.412892 3798194 memory.go:191] Add success. I1121 21:01:26.417067 3798194 cpu.go:282] Add success. I1121 21:01:26.441665 3798194 net.go:642] Add success. I1121 21:01:26.443224 3798194 net.go:778] primary dev: ETH0 I1121 21:01:26.443241 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:01:26.443253 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:01:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:01:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:01:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:01:36.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:01:36.412265 3798194 memory.go:184] no items to output this cycle I1121 21:01:36.412408 3798194 cpu.go:275] no items to output this cycle E1121 21:01:46.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:01:46.412262 3798194 memory.go:184] no items to output this cycle I1121 21:01:46.412409 3798194 cpu.go:275] no items to output this cycle E1121 21:01:56.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:01:56.411952 3798194 memory.go:191] Add success. W1121 21:01:56.411981 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:01:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:01:56.411996 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:01:56.412351 3798194 cpu.go:282] Add success. I1121 21:01:56.441945 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:01:56.442883 3798194 net.go:642] Add success. I1121 21:01:56.447104 3798194 net.go:778] primary dev: ETH0 I1121 21:01:56.447130 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:01:56.447143 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:01:57.450590 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:01:57.450787 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:01:57.450797 3798194 disk_worker.go:723] disk space is not compliant W1121 21:01:57.450800 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:01:57.459102 3798194 disk_worker.go:509] system disk:vda1 I1121 21:01:57.459159 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:01:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:01:59.450010 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:01:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:01:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:01:59.471636 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:01:59.654143 3798194 disk_info.go:125] begin check local disk info of client I1121 21:01:59.658020 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:01:59.658029 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000480d40 0xc000480d80] E1121 21:02:06.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:02:06.413253 3798194 memory.go:184] no items to output this cycle I1121 21:02:06.413408 3798194 cpu.go:275] no items to output this cycle E1121 21:02:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:02:16.411980 3798194 memory.go:184] no items to output this cycle I1121 21:02:16.412142 3798194 cpu.go:275] no items to output this cycle E1121 21:02:26.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:02:26.411958 3798194 memory.go:191] Add success. I1121 21:02:26.412314 3798194 cpu.go:282] Add success. I1121 21:02:26.441741 3798194 net.go:642] Add success. I1121 21:02:26.445889 3798194 net.go:778] primary dev: ETH0 I1121 21:02:26.445905 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:02:26.445919 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:02:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:02:29.450086 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:02:29.450119 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:02:36.413166 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:02:36.413188 3798194 memory.go:184] no items to output this cycle I1121 21:02:36.413330 3798194 cpu.go:275] no items to output this cycle E1121 21:02:46.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:02:46.412260 3798194 memory.go:184] no items to output this cycle I1121 21:02:46.412405 3798194 cpu.go:275] no items to output this cycle E1121 21:02:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:02:56.411953 3798194 memory.go:191] Add success. W1121 21:02:56.411981 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:02:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:02:56.411996 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:02:56.412348 3798194 cpu.go:282] Add success. I1121 21:02:56.441806 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:02:56.442330 3798194 net.go:642] Add success. I1121 21:02:56.446417 3798194 net.go:778] primary dev: ETH0 I1121 21:02:56.446432 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:02:56.446445 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:02:57.450796 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:02:57.450990 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:02:57.451001 3798194 disk_worker.go:723] disk space is not compliant W1121 21:02:57.451004 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:02:57.460139 3798194 disk_worker.go:509] system disk:vda1 I1121 21:02:57.460182 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:02:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:02:59.450101 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:02:59.450202 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:02:59.450228 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:02:59.470708 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:02:59.658161 3798194 disk_info.go:125] begin check local disk info of client I1121 21:02:59.661996 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:02:59.662005 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fe900 0xc0003fe940] E1121 21:03:06.413154 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:03:06.413175 3798194 memory.go:184] no items to output this cycle I1121 21:03:06.413323 3798194 cpu.go:275] no items to output this cycle E1121 21:03:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:03:16.411989 3798194 memory.go:184] no items to output this cycle I1121 21:03:16.412151 3798194 cpu.go:275] no items to output this cycle I1121 21:03:17.555443 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:03:17.555452 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:03:26.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:03:26.412885 3798194 memory.go:191] Add success. I1121 21:03:26.417302 3798194 cpu.go:282] Add success. I1121 21:03:26.441796 3798194 net.go:642] Add success. I1121 21:03:26.443309 3798194 net.go:778] primary dev: ETH0 I1121 21:03:26.443325 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:03:26.443338 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:03:29.450508 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:03:29.450583 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:03:29.450613 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:03:36.412872 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:03:36.412900 3798194 memory.go:184] no items to output this cycle I1121 21:03:36.413054 3798194 cpu.go:275] no items to output this cycle E1121 21:03:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:03:46.411982 3798194 memory.go:184] no items to output this cycle I1121 21:03:46.412141 3798194 cpu.go:275] no items to output this cycle E1121 21:03:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:03:56.411950 3798194 memory.go:191] Add success. W1121 21:03:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:03:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:03:56.412002 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:03:56.412355 3798194 cpu.go:282] Add success. I1121 21:03:56.441977 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:03:56.442594 3798194 net.go:642] Add success. I1121 21:03:56.446916 3798194 net.go:778] primary dev: ETH0 I1121 21:03:56.446931 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:03:56.446943 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:03:57.451711 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:03:57.451886 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:03:57.451939 3798194 disk_worker.go:723] disk space is not compliant W1121 21:03:57.451942 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:03:57.460333 3798194 disk_worker.go:509] system disk:vda1 I1121 21:03:57.460372 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:03:58.445997 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:03:59.450098 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:03:59.450189 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:03:59.450218 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:03:59.470708 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:03:59.662106 3798194 disk_info.go:125] begin check local disk info of client I1121 21:03:59.666079 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:03:59.666089 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000368380 0xc0003683c0] E1121 21:04:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:04:06.412255 3798194 memory.go:184] no items to output this cycle I1121 21:04:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 21:04:16.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:04:16.413263 3798194 memory.go:184] no items to output this cycle I1121 21:04:16.413407 3798194 cpu.go:275] no items to output this cycle E1121 21:04:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:04:26.411946 3798194 memory.go:191] Add success. I1121 21:04:26.412302 3798194 cpu.go:282] Add success. I1121 21:04:26.441745 3798194 net.go:642] Add success. I1121 21:04:26.446159 3798194 net.go:778] primary dev: ETH0 I1121 21:04:26.446176 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:04:26.446189 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:04:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:04:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:04:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:04:36.412087 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:04:36.412126 3798194 memory.go:184] no items to output this cycle I1121 21:04:36.412384 3798194 cpu.go:275] no items to output this cycle E1121 21:04:46.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:04:46.413255 3798194 memory.go:184] no items to output this cycle I1121 21:04:46.413402 3798194 cpu.go:275] no items to output this cycle E1121 21:04:56.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:04:56.413272 3798194 memory.go:191] Add success. W1121 21:04:56.413300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:04:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:04:56.413315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:04:56.417711 3798194 cpu.go:282] Add success. I1121 21:04:56.441250 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:04:56.441720 3798194 net.go:642] Add success. I1121 21:04:56.443371 3798194 net.go:778] primary dev: ETH0 I1121 21:04:56.443387 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:04:56.443401 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:04:57.446455 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:04:57.446709 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:04:57.446721 3798194 disk_worker.go:723] disk space is not compliant W1121 21:04:57.446724 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:04:57.449503 3798194 disk_worker.go:509] system disk:vda1 I1121 21:04:57.449544 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:04:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:04:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:04:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:04:59.450503 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:04:59.471067 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:04:59.666154 3798194 disk_info.go:125] begin check local disk info of client I1121 21:04:59.669946 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:04:59.669956 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041c800 0xc00041c840] E1121 21:05:06.412639 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:05:06.412663 3798194 memory.go:184] no items to output this cycle I1121 21:05:06.412807 3798194 cpu.go:275] no items to output this cycle E1121 21:05:16.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:05:16.412878 3798194 memory.go:184] no items to output this cycle I1121 21:05:16.413023 3798194 cpu.go:275] no items to output this cycle E1121 21:05:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:05:26.411958 3798194 memory.go:191] Add success. I1121 21:05:26.412302 3798194 cpu.go:282] Add success. I1121 21:05:26.441905 3798194 net.go:642] Add success. I1121 21:05:26.446067 3798194 net.go:778] primary dev: ETH0 I1121 21:05:26.446084 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:05:26.446098 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:05:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:05:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:05:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:05:36.412775 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:05:36.412809 3798194 memory.go:184] no items to output this cycle I1121 21:05:36.412969 3798194 cpu.go:275] no items to output this cycle E1121 21:05:46.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:05:46.412268 3798194 memory.go:184] no items to output this cycle I1121 21:05:46.412418 3798194 cpu.go:275] no items to output this cycle E1121 21:05:56.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:05:56.413267 3798194 memory.go:191] Add success. W1121 21:05:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:05:56.413316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:05:56.413319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:05:56.417885 3798194 cpu.go:282] Add success. I1121 21:05:56.442225 3798194 event_worker.go:152] Polling the log file for events... I1121 21:05:56.442597 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:05:56.443098 3798194 net.go:642] Add success. I1121 21:05:56.449443 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795264556, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 21:05:56.449550 3798194 net.go:778] primary dev: ETH0 I1121 21:05:56.449564 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:05:56.449576 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:05:56.455381 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 21:05:57.448237 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 21:05:57.448252 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:05:57.448457 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:05:57.448522 3798194 disk_worker.go:723] disk space is not compliant W1121 21:05:57.448525 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:05:57.452224 3798194 disk_worker.go:509] system disk:vda1 I1121 21:05:57.452264 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 21:05:58.448496 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 21:05:58.448517 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 21:05:59.449313 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 21:05:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:05:59.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:05:59.450504 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:05:59.477198 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:05:59.670057 3798194 disk_info.go:125] begin check local disk info of client I1121 21:05:59.673877 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:05:59.673886 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e7c0 0xc00030e800] E1121 21:06:06.412940 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:06:06.412963 3798194 memory.go:184] no items to output this cycle I1121 21:06:06.413135 3798194 cpu.go:275] no items to output this cycle E1121 21:06:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:06:16.412262 3798194 memory.go:184] no items to output this cycle I1121 21:06:16.412411 3798194 cpu.go:275] no items to output this cycle I1121 21:06:17.555614 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:06:17.555624 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:06:26.411197 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:06:26.411230 3798194 memory.go:191] Add success. I1121 21:06:26.411567 3798194 cpu.go:282] Add success. I1121 21:06:26.441275 3798194 net.go:642] Add success. I1121 21:06:26.445218 3798194 net.go:778] primary dev: ETH0 I1121 21:06:26.445233 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:06:26.445246 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:06:29.450004 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:06:29.450086 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:06:29.450123 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:06:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:06:36.411980 3798194 memory.go:184] no items to output this cycle I1121 21:06:36.412136 3798194 cpu.go:275] no items to output this cycle E1121 21:06:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:06:46.412253 3798194 memory.go:184] no items to output this cycle I1121 21:06:46.412399 3798194 cpu.go:275] no items to output this cycle E1121 21:06:56.412847 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:06:56.412878 3798194 memory.go:191] Add success. W1121 21:06:56.412907 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:06:56.412918 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:06:56.412923 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:06:56.417273 3798194 cpu.go:282] Add success. I1121 21:06:56.441658 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:06:56.442160 3798194 net.go:642] Add success. I1121 21:06:56.443514 3798194 net.go:778] primary dev: ETH0 I1121 21:06:56.443529 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:06:56.443542 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:06:57.450911 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:06:57.451139 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:06:57.451151 3798194 disk_worker.go:723] disk space is not compliant W1121 21:06:57.451155 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:06:57.459873 3798194 disk_worker.go:509] system disk:vda1 I1121 21:06:57.459917 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:06:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:06:59.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:06:59.450465 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:06:59.450494 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:06:59.471010 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:06:59.673975 3798194 disk_info.go:125] begin check local disk info of client I1121 21:06:59.677794 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:06:59.677803 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003dbe80 0xc0003dbec0] E1121 21:07:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:07:06.411984 3798194 memory.go:184] no items to output this cycle I1121 21:07:06.412137 3798194 cpu.go:275] no items to output this cycle E1121 21:07:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:07:16.413264 3798194 memory.go:184] no items to output this cycle I1121 21:07:16.413411 3798194 cpu.go:275] no items to output this cycle E1121 21:07:26.413254 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:07:26.413289 3798194 memory.go:191] Add success. I1121 21:07:26.417444 3798194 cpu.go:282] Add success. I1121 21:07:26.441937 3798194 net.go:642] Add success. I1121 21:07:26.443457 3798194 net.go:778] primary dev: ETH0 I1121 21:07:26.443473 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:07:26.443487 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:07:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:07:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:07:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:07:36.412463 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:07:36.412485 3798194 memory.go:184] no items to output this cycle I1121 21:07:36.412628 3798194 cpu.go:275] no items to output this cycle E1121 21:07:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:07:46.412255 3798194 memory.go:184] no items to output this cycle I1121 21:07:46.412401 3798194 cpu.go:275] no items to output this cycle E1121 21:07:56.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:07:56.412264 3798194 memory.go:191] Add success. W1121 21:07:56.412296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:07:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:07:56.412312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:07:56.416564 3798194 cpu.go:282] Add success. I1121 21:07:56.440936 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:07:56.441503 3798194 net.go:642] Add success. I1121 21:07:56.442962 3798194 net.go:778] primary dev: ETH0 I1121 21:07:56.442979 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:07:56.442992 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:07:57.449602 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:07:57.449842 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:07:57.449855 3798194 disk_worker.go:723] disk space is not compliant W1121 21:07:57.449858 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:07:57.451709 3798194 disk_worker.go:509] system disk:vda1 I1121 21:07:57.451754 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:07:58.446043 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:07:59.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:07:59.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:07:59.450152 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:07:59.471703 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:07:59.678174 3798194 disk_info.go:125] begin check local disk info of client I1121 21:07:59.683587 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:07:59.683598 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000364400 0xc000364440] E1121 21:08:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:08:06.411982 3798194 memory.go:184] no items to output this cycle I1121 21:08:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 21:08:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:08:16.411973 3798194 memory.go:184] no items to output this cycle I1121 21:08:16.412135 3798194 cpu.go:275] no items to output this cycle E1121 21:08:26.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:08:26.413269 3798194 memory.go:191] Add success. I1121 21:08:26.417686 3798194 cpu.go:282] Add success. I1121 21:08:26.442239 3798194 net.go:642] Add success. I1121 21:08:26.443967 3798194 net.go:778] primary dev: ETH0 I1121 21:08:26.443982 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:08:26.443993 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:08:29.449627 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:08:29.449702 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:08:29.449727 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:08:36.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:08:36.411975 3798194 memory.go:184] no items to output this cycle I1121 21:08:36.412252 3798194 cpu.go:275] no items to output this cycle E1121 21:08:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:08:46.412254 3798194 memory.go:184] no items to output this cycle I1121 21:08:46.412400 3798194 cpu.go:275] no items to output this cycle E1121 21:08:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:08:56.411950 3798194 memory.go:191] Add success. W1121 21:08:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:08:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:08:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:08:56.412382 3798194 cpu.go:282] Add success. I1121 21:08:56.441832 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:08:56.442433 3798194 net.go:642] Add success. I1121 21:08:56.446634 3798194 net.go:778] primary dev: ETH0 I1121 21:08:56.446651 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:08:56.446663 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:08:57.450578 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:08:57.450736 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:08:57.450787 3798194 disk_worker.go:723] disk space is not compliant W1121 21:08:57.450791 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:08:57.459095 3798194 disk_worker.go:509] system disk:vda1 I1121 21:08:57.459150 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:08:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:08:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:08:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:08:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:08:59.471660 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:08:59.683701 3798194 disk_info.go:125] begin check local disk info of client I1121 21:08:59.687652 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:08:59.687664 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041c880 0xc00041c8c0] E1121 21:09:06.412775 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:09:06.412796 3798194 memory.go:184] no items to output this cycle I1121 21:09:06.412955 3798194 cpu.go:275] no items to output this cycle E1121 21:09:16.412994 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:09:16.413016 3798194 memory.go:184] no items to output this cycle I1121 21:09:16.413181 3798194 cpu.go:275] no items to output this cycle I1121 21:09:17.555785 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:09:17.555794 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:09:26.411995 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:09:26.412031 3798194 memory.go:191] Add success. I1121 21:09:26.416475 3798194 cpu.go:282] Add success. I1121 21:09:26.441979 3798194 net.go:642] Add success. I1121 21:09:26.448365 3798194 net.go:778] primary dev: ETH0 I1121 21:09:26.448380 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:09:26.448395 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:09:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:09:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:09:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:09:36.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:09:36.411972 3798194 memory.go:184] no items to output this cycle I1121 21:09:36.412120 3798194 cpu.go:275] no items to output this cycle E1121 21:09:46.413289 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:09:46.413312 3798194 memory.go:184] no items to output this cycle I1121 21:09:46.413458 3798194 cpu.go:275] no items to output this cycle E1121 21:09:56.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:09:56.413264 3798194 memory.go:191] Add success. W1121 21:09:56.413292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:09:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:09:56.413309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:09:56.418033 3798194 cpu.go:282] Add success. I1121 21:09:56.441432 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:09:56.441905 3798194 net.go:642] Add success. I1121 21:09:56.443484 3798194 net.go:778] primary dev: ETH0 I1121 21:09:56.443498 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:09:56.443510 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:09:57.445475 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:09:57.445675 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:09:57.445740 3798194 disk_worker.go:723] disk space is not compliant W1121 21:09:57.445743 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:09:57.447849 3798194 disk_worker.go:509] system disk:vda1 I1121 21:09:57.447888 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:09:58.446374 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:09:59.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:09:59.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:09:59.450128 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:09:59.471656 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:09:59.687763 3798194 disk_info.go:125] begin check local disk info of client I1121 21:09:59.691559 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:09:59.691568 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000499740 0xc000499780] E1121 21:10:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:10:06.412260 3798194 memory.go:184] no items to output this cycle I1121 21:10:06.412403 3798194 cpu.go:275] no items to output this cycle E1121 21:10:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:10:16.412257 3798194 memory.go:184] no items to output this cycle I1121 21:10:16.412406 3798194 cpu.go:275] no items to output this cycle E1121 21:10:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:10:26.413266 3798194 memory.go:191] Add success. I1121 21:10:26.417597 3798194 cpu.go:282] Add success. I1121 21:10:26.442130 3798194 net.go:642] Add success. I1121 21:10:26.443660 3798194 net.go:778] primary dev: ETH0 I1121 21:10:26.443675 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:10:26.443688 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:10:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:10:29.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:10:29.450511 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:10:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:10:36.411984 3798194 memory.go:184] no items to output this cycle I1121 21:10:36.412148 3798194 cpu.go:275] no items to output this cycle E1121 21:10:46.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:10:46.412248 3798194 memory.go:184] no items to output this cycle I1121 21:10:46.412397 3798194 cpu.go:275] no items to output this cycle E1121 21:10:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:10:56.411953 3798194 memory.go:191] Add success. W1121 21:10:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:10:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:10:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:10:56.412349 3798194 cpu.go:282] Add success. I1121 21:10:56.445667 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:10:56.446182 3798194 net.go:642] Add success. I1121 21:10:56.450465 3798194 net.go:778] primary dev: ETH0 I1121 21:10:56.450481 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:10:56.450493 3798194 net.go:659] Add eni metrics success, count: 7 E1121 21:10:57.445450 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 21:10:57.445465 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:10:57.445658 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:10:57.445729 3798194 disk_worker.go:723] disk space is not compliant W1121 21:10:57.445732 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:10:57.447843 3798194 disk_worker.go:509] system disk:vda1 I1121 21:10:57.447883 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 21:10:58.448088 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 21:10:58.448103 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 21:10:59.448621 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 21:10:59.449662 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:10:59.449714 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:10:59.449742 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:10:59.471185 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:10:59.691666 3798194 disk_info.go:125] begin check local disk info of client I1121 21:10:59.695634 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:10:59.695644 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fa540 0xc0003fa580] E1121 21:11:06.412950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:11:06.412973 3798194 memory.go:184] no items to output this cycle I1121 21:11:06.413130 3798194 cpu.go:275] no items to output this cycle E1121 21:11:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:11:16.413258 3798194 memory.go:184] no items to output this cycle I1121 21:11:16.413408 3798194 cpu.go:275] no items to output this cycle E1121 21:11:26.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:11:26.411943 3798194 memory.go:191] Add success. I1121 21:11:26.412289 3798194 cpu.go:282] Add success. I1121 21:11:26.442219 3798194 net.go:642] Add success. I1121 21:11:26.446443 3798194 net.go:778] primary dev: ETH0 I1121 21:11:26.446463 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:11:26.446481 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:11:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:11:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:11:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:11:36.412845 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:11:36.412869 3798194 memory.go:184] no items to output this cycle I1121 21:11:36.413011 3798194 cpu.go:275] no items to output this cycle E1121 21:11:46.412588 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:11:46.412609 3798194 memory.go:184] no items to output this cycle I1121 21:11:46.412769 3798194 cpu.go:275] no items to output this cycle E1121 21:11:56.412879 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:11:56.412913 3798194 memory.go:191] Add success. W1121 21:11:56.412942 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:11:56.412955 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:11:56.412960 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:11:56.417264 3798194 cpu.go:282] Add success. I1121 21:11:56.441665 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:11:56.442187 3798194 net.go:642] Add success. I1121 21:11:56.443723 3798194 net.go:778] primary dev: ETH0 I1121 21:11:56.443738 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:11:56.443750 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:11:57.450581 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:11:57.450782 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:11:57.450793 3798194 disk_worker.go:723] disk space is not compliant W1121 21:11:57.450796 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:11:57.459175 3798194 disk_worker.go:509] system disk:vda1 I1121 21:11:57.459228 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:11:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:11:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:11:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:11:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:11:59.471691 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:11:59.695860 3798194 disk_info.go:125] begin check local disk info of client I1121 21:11:59.699724 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:11:59.699733 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046a980 0xc00046a9c0] E1121 21:12:06.411977 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:12:06.412002 3798194 memory.go:184] no items to output this cycle I1121 21:12:06.412166 3798194 cpu.go:275] no items to output this cycle E1121 21:12:16.412249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:12:16.412272 3798194 memory.go:184] no items to output this cycle I1121 21:12:16.412416 3798194 cpu.go:275] no items to output this cycle I1121 21:12:17.556319 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:12:17.556328 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:12:26.411224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:12:26.411257 3798194 memory.go:191] Add success. I1121 21:12:26.411610 3798194 cpu.go:282] Add success. I1121 21:12:26.442200 3798194 net.go:642] Add success. I1121 21:12:26.446667 3798194 net.go:778] primary dev: ETH0 I1121 21:12:26.446682 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:12:26.446695 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:12:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:12:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:12:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:12:36.412811 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:12:36.412834 3798194 memory.go:184] no items to output this cycle I1121 21:12:36.412986 3798194 cpu.go:275] no items to output this cycle E1121 21:12:46.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:12:46.412249 3798194 memory.go:184] no items to output this cycle I1121 21:12:46.412406 3798194 cpu.go:275] no items to output this cycle E1121 21:12:56.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:12:56.412269 3798194 memory.go:191] Add success. W1121 21:12:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:12:56.412315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:12:56.412318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:12:56.416726 3798194 cpu.go:282] Add success. I1121 21:12:56.441155 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:12:56.442436 3798194 net.go:642] Add success. I1121 21:12:56.444036 3798194 net.go:778] primary dev: ETH0 I1121 21:12:56.444051 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:12:56.444069 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:12:57.451224 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:12:57.451443 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:12:57.451453 3798194 disk_worker.go:723] disk space is not compliant W1121 21:12:57.451457 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:12:57.460762 3798194 disk_worker.go:509] system disk:vda1 I1121 21:12:57.460804 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:12:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:12:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:12:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:12:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:12:59.471664 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:12:59.699831 3798194 disk_info.go:125] begin check local disk info of client I1121 21:12:59.703718 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:12:59.703727 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004f6280 0xc0004f62c0] E1121 21:13:06.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:13:06.411981 3798194 memory.go:184] no items to output this cycle I1121 21:13:06.412145 3798194 cpu.go:275] no items to output this cycle E1121 21:13:16.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:13:16.413264 3798194 memory.go:184] no items to output this cycle I1121 21:13:16.413413 3798194 cpu.go:275] no items to output this cycle E1121 21:13:26.411934 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:13:26.411968 3798194 memory.go:191] Add success. I1121 21:13:26.412293 3798194 cpu.go:282] Add success. I1121 21:13:26.441751 3798194 net.go:642] Add success. I1121 21:13:26.445578 3798194 net.go:778] primary dev: ETH0 I1121 21:13:26.445593 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:13:26.445606 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:13:29.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:13:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:13:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:13:36.413220 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:13:36.413242 3798194 memory.go:184] no items to output this cycle I1121 21:13:36.413394 3798194 cpu.go:275] no items to output this cycle E1121 21:13:46.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:13:46.413269 3798194 memory.go:184] no items to output this cycle I1121 21:13:46.413417 3798194 cpu.go:275] no items to output this cycle E1121 21:13:56.413038 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:13:56.413073 3798194 memory.go:191] Add success. W1121 21:13:56.413122 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:13:56.413140 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:13:56.413144 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:13:56.417683 3798194 cpu.go:282] Add success. I1121 21:13:56.441074 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:13:56.441563 3798194 net.go:642] Add success. I1121 21:13:56.443173 3798194 net.go:778] primary dev: ETH0 I1121 21:13:56.443187 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:13:56.443199 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:13:57.450636 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:13:57.450792 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:13:57.450845 3798194 disk_worker.go:723] disk space is not compliant W1121 21:13:57.450848 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:13:57.459201 3798194 disk_worker.go:509] system disk:vda1 I1121 21:13:57.459245 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:13:58.446017 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:13:59.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:13:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:13:59.450146 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:13:59.471676 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:13:59.704142 3798194 disk_info.go:125] begin check local disk info of client I1121 21:13:59.709190 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:13:59.709201 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d68c0 0xc0003d6900] E1121 21:14:06.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:14:06.412881 3798194 memory.go:184] no items to output this cycle I1121 21:14:06.413020 3798194 cpu.go:275] no items to output this cycle E1121 21:14:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:14:16.411986 3798194 memory.go:184] no items to output this cycle I1121 21:14:16.412149 3798194 cpu.go:275] no items to output this cycle E1121 21:14:26.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:14:26.411944 3798194 memory.go:191] Add success. I1121 21:14:26.412332 3798194 cpu.go:282] Add success. I1121 21:14:26.441930 3798194 net.go:642] Add success. I1121 21:14:26.446242 3798194 net.go:778] primary dev: ETH0 I1121 21:14:26.446259 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:14:26.446272 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:14:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:14:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:14:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:14:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:14:36.412256 3798194 memory.go:184] no items to output this cycle I1121 21:14:36.412403 3798194 cpu.go:275] no items to output this cycle E1121 21:14:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:14:46.412256 3798194 memory.go:184] no items to output this cycle I1121 21:14:46.412399 3798194 cpu.go:275] no items to output this cycle E1121 21:14:56.412322 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:14:56.412358 3798194 memory.go:191] Add success. W1121 21:14:56.412390 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:14:56.412402 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:14:56.412407 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:14:56.416720 3798194 cpu.go:282] Add success. I1121 21:14:56.441099 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:14:56.441659 3798194 net.go:642] Add success. I1121 21:14:56.443321 3798194 net.go:778] primary dev: ETH0 I1121 21:14:56.443336 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:14:56.443349 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:14:57.450919 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:14:57.451145 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:14:57.451156 3798194 disk_worker.go:723] disk space is not compliant W1121 21:14:57.451159 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:14:57.460426 3798194 disk_worker.go:509] system disk:vda1 I1121 21:14:57.460468 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:14:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:14:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:14:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:14:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:14:59.471672 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:14:59.709303 3798194 disk_info.go:125] begin check local disk info of client I1121 21:14:59.713183 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:14:59.713192 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000342340 0xc000342380] E1121 21:15:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:15:06.411983 3798194 memory.go:184] no items to output this cycle I1121 21:15:06.412138 3798194 cpu.go:275] no items to output this cycle E1121 21:15:16.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:15:16.412253 3798194 memory.go:184] no items to output this cycle I1121 21:15:16.412398 3798194 cpu.go:275] no items to output this cycle I1121 21:15:17.556497 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:15:17.556507 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:15:26.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:15:26.413275 3798194 memory.go:191] Add success. I1121 21:15:26.417655 3798194 cpu.go:282] Add success. I1121 21:15:26.442304 3798194 net.go:642] Add success. I1121 21:15:26.443920 3798194 net.go:778] primary dev: ETH0 I1121 21:15:26.443935 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:15:26.443948 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:15:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:15:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:15:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:15:36.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:15:36.412251 3798194 memory.go:184] no items to output this cycle I1121 21:15:36.412392 3798194 cpu.go:275] no items to output this cycle E1121 21:15:46.413256 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:15:46.413282 3798194 memory.go:184] no items to output this cycle I1121 21:15:46.413432 3798194 cpu.go:275] no items to output this cycle E1121 21:15:56.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:15:56.413266 3798194 memory.go:191] Add success. W1121 21:15:56.413293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:15:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:15:56.413310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:15:56.417685 3798194 cpu.go:282] Add success. I1121 21:15:56.443743 3798194 event_worker.go:152] Polling the log file for events... I1121 21:15:56.443995 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:15:56.444522 3798194 net.go:642] Add success. I1121 21:15:56.446078 3798194 net.go:778] primary dev: ETH0 I1121 21:15:56.446092 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:15:56.446103 3798194 net.go:659] Add eni metrics success, count: 7 E1121 21:15:57.450710 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 21:15:57.450723 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:15:57.450880 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:15:57.450928 3798194 disk_worker.go:723] disk space is not compliant W1121 21:15:57.450931 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:15:57.459138 3798194 disk_worker.go:509] system disk:vda1 I1121 21:15:57.459179 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 21:15:58.447880 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 21:15:58.447894 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 21:15:59.448536 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 21:15:59.449617 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:15:59.449669 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:15:59.449691 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:15:59.471219 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:15:59.713297 3798194 disk_info.go:125] begin check local disk info of client I1121 21:15:59.717057 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:15:59.717066 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000410700 0xc000410740] E1121 21:16:06.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:16:06.411988 3798194 memory.go:184] no items to output this cycle I1121 21:16:06.412145 3798194 cpu.go:275] no items to output this cycle E1121 21:16:16.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:16:16.413257 3798194 memory.go:184] no items to output this cycle I1121 21:16:16.413414 3798194 cpu.go:275] no items to output this cycle E1121 21:16:26.413060 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:16:26.413095 3798194 memory.go:191] Add success. I1121 21:16:26.417268 3798194 cpu.go:282] Add success. I1121 21:16:26.441758 3798194 net.go:642] Add success. I1121 21:16:26.443247 3798194 net.go:778] primary dev: ETH0 I1121 21:16:26.443262 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:16:26.443276 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:16:29.449954 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:16:29.450038 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:16:29.450064 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:16:36.412738 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:16:36.412760 3798194 memory.go:184] no items to output this cycle I1121 21:16:36.412904 3798194 cpu.go:275] no items to output this cycle E1121 21:16:46.412845 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:16:46.412866 3798194 memory.go:184] no items to output this cycle I1121 21:16:46.413011 3798194 cpu.go:275] no items to output this cycle E1121 21:16:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:16:56.411951 3798194 memory.go:191] Add success. W1121 21:16:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:16:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:16:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:16:56.412354 3798194 cpu.go:282] Add success. I1121 21:16:56.441895 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:16:56.442451 3798194 net.go:642] Add success. I1121 21:16:56.446311 3798194 net.go:778] primary dev: ETH0 I1121 21:16:56.446331 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:16:56.446345 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:16:57.450259 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:16:57.450653 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:16:57.450668 3798194 disk_worker.go:723] disk space is not compliant W1121 21:16:57.450671 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:16:57.459279 3798194 disk_worker.go:509] system disk:vda1 I1121 21:16:57.459319 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:16:58.446009 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:16:59.450116 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:16:59.450201 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:16:59.450226 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:16:59.470692 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:16:59.717155 3798194 disk_info.go:125] begin check local disk info of client I1121 21:16:59.720949 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:16:59.720957 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a1800 0xc0004a1840] E1121 21:17:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:17:06.412255 3798194 memory.go:184] no items to output this cycle I1121 21:17:06.412408 3798194 cpu.go:275] no items to output this cycle E1121 21:17:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:17:16.411970 3798194 memory.go:184] no items to output this cycle I1121 21:17:16.412124 3798194 cpu.go:275] no items to output this cycle E1121 21:17:26.413407 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:17:26.413442 3798194 memory.go:191] Add success. I1121 21:17:26.417493 3798194 cpu.go:282] Add success. I1121 21:17:26.441995 3798194 net.go:642] Add success. I1121 21:17:26.443506 3798194 net.go:778] primary dev: ETH0 I1121 21:17:26.443521 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:17:26.443534 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:17:29.450309 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:17:29.450386 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:17:29.450412 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:17:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:17:36.412258 3798194 memory.go:184] no items to output this cycle I1121 21:17:36.412401 3798194 cpu.go:275] no items to output this cycle E1121 21:17:46.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:17:46.411988 3798194 memory.go:184] no items to output this cycle I1121 21:17:46.412146 3798194 cpu.go:275] no items to output this cycle E1121 21:17:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:17:56.411950 3798194 memory.go:191] Add success. W1121 21:17:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:17:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:17:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:17:56.412358 3798194 cpu.go:282] Add success. I1121 21:17:56.441820 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:17:56.442435 3798194 net.go:642] Add success. I1121 21:17:56.446593 3798194 net.go:778] primary dev: ETH0 I1121 21:17:56.446609 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:17:56.446622 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:17:57.450549 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:17:57.450754 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:17:57.450766 3798194 disk_worker.go:723] disk space is not compliant W1121 21:17:57.450769 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:17:57.459056 3798194 disk_worker.go:509] system disk:vda1 I1121 21:17:57.459099 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:17:58.445994 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:17:59.450163 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:17:59.450235 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:17:59.450269 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:17:59.470940 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:17:59.721141 3798194 disk_info.go:125] begin check local disk info of client I1121 21:17:59.724882 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:17:59.724891 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d4c00 0xc0003d4c40] E1121 21:18:06.412085 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:18:06.412124 3798194 memory.go:184] no items to output this cycle I1121 21:18:06.412269 3798194 cpu.go:275] no items to output this cycle E1121 21:18:16.413158 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:18:16.413182 3798194 memory.go:184] no items to output this cycle I1121 21:18:16.413330 3798194 cpu.go:275] no items to output this cycle I1121 21:18:17.557647 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:18:17.557656 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:18:26.412679 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:18:26.412717 3798194 memory.go:191] Add success. I1121 21:18:26.417021 3798194 cpu.go:282] Add success. I1121 21:18:26.441542 3798194 net.go:642] Add success. I1121 21:18:26.443149 3798194 net.go:778] primary dev: ETH0 I1121 21:18:26.443164 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:18:26.443177 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:18:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:18:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:18:29.450502 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:18:36.412532 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:18:36.412557 3798194 memory.go:184] no items to output this cycle I1121 21:18:36.412699 3798194 cpu.go:275] no items to output this cycle E1121 21:18:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:18:46.411972 3798194 memory.go:184] no items to output this cycle I1121 21:18:46.412133 3798194 cpu.go:275] no items to output this cycle E1121 21:18:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:18:56.411949 3798194 memory.go:191] Add success. W1121 21:18:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:18:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:18:56.411991 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:18:56.412355 3798194 cpu.go:282] Add success. I1121 21:18:56.441827 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:18:56.442433 3798194 net.go:642] Add success. I1121 21:18:56.446460 3798194 net.go:778] primary dev: ETH0 I1121 21:18:56.446479 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:18:56.446496 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:18:57.450648 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:18:57.450815 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:18:57.450865 3798194 disk_worker.go:723] disk space is not compliant W1121 21:18:57.450868 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:18:57.459202 3798194 disk_worker.go:509] system disk:vda1 I1121 21:18:57.459242 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:18:58.445996 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:18:59.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:18:59.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:18:59.450507 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:18:59.471034 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:18:59.725143 3798194 disk_info.go:125] begin check local disk info of client I1121 21:18:59.728995 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:18:59.729005 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000301400 0xc000301440] E1121 21:19:06.413171 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:19:06.413195 3798194 memory.go:184] no items to output this cycle I1121 21:19:06.413336 3798194 cpu.go:275] no items to output this cycle E1121 21:19:16.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:19:16.412249 3798194 memory.go:184] no items to output this cycle I1121 21:19:16.412399 3798194 cpu.go:275] no items to output this cycle E1121 21:19:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:19:26.411952 3798194 memory.go:191] Add success. I1121 21:19:26.412304 3798194 cpu.go:282] Add success. I1121 21:19:26.441748 3798194 net.go:642] Add success. I1121 21:19:26.446045 3798194 net.go:778] primary dev: ETH0 I1121 21:19:26.446061 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:19:26.446079 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:19:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:19:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:19:29.450502 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:19:36.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:19:36.413267 3798194 memory.go:184] no items to output this cycle I1121 21:19:36.413420 3798194 cpu.go:275] no items to output this cycle E1121 21:19:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:19:46.411982 3798194 memory.go:184] no items to output this cycle I1121 21:19:46.412256 3798194 cpu.go:275] no items to output this cycle E1121 21:19:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:19:56.411952 3798194 memory.go:191] Add success. W1121 21:19:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:19:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:19:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:19:56.412370 3798194 cpu.go:282] Add success. I1121 21:19:56.441832 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:19:56.442335 3798194 net.go:642] Add success. I1121 21:19:56.446343 3798194 net.go:778] primary dev: ETH0 I1121 21:19:56.446359 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:19:56.446372 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:19:57.450983 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:19:57.451222 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:19:57.451234 3798194 disk_worker.go:723] disk space is not compliant W1121 21:19:57.451237 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:19:57.460062 3798194 disk_worker.go:509] system disk:vda1 I1121 21:19:57.460105 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:19:58.445997 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:19:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:19:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:19:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:19:59.470693 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:19:59.734184 3798194 disk_info.go:125] begin check local disk info of client I1121 21:19:59.739718 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:19:59.739730 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0001bedc0 0xc0001bee00] E1121 21:20:06.412762 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:20:06.412783 3798194 memory.go:184] no items to output this cycle I1121 21:20:06.412940 3798194 cpu.go:275] no items to output this cycle E1121 21:20:16.413270 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:20:16.413294 3798194 memory.go:184] no items to output this cycle I1121 21:20:16.413449 3798194 cpu.go:275] no items to output this cycle E1121 21:20:26.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:20:26.413259 3798194 memory.go:191] Add success. I1121 21:20:26.417591 3798194 cpu.go:282] Add success. I1121 21:20:26.442134 3798194 net.go:642] Add success. I1121 21:20:26.443721 3798194 net.go:778] primary dev: ETH0 I1121 21:20:26.443735 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:20:26.443749 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:20:29.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:20:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:20:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:20:36.413051 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:20:36.413074 3798194 memory.go:184] no items to output this cycle I1121 21:20:36.413244 3798194 cpu.go:275] no items to output this cycle E1121 21:20:46.411936 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:20:46.411959 3798194 memory.go:184] no items to output this cycle I1121 21:20:46.412100 3798194 cpu.go:275] no items to output this cycle E1121 21:20:56.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:20:56.413262 3798194 memory.go:191] Add success. W1121 21:20:56.413289 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:20:56.413300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:20:56.413304 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:20:56.417532 3798194 cpu.go:282] Add success. I1121 21:20:56.444830 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:20:56.445358 3798194 net.go:642] Add success. I1121 21:20:56.449725 3798194 net.go:778] primary dev: ETH0 I1121 21:20:56.449739 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:20:56.449753 3798194 net.go:659] Add eni metrics success, count: 7 E1121 21:20:57.450956 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 21:20:57.450970 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:20:57.451177 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:20:57.451188 3798194 disk_worker.go:723] disk space is not compliant W1121 21:20:57.451191 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:20:57.460704 3798194 disk_worker.go:509] system disk:vda1 I1121 21:20:57.460742 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 21:20:58.448466 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 21:20:58.448481 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 21:20:59.448826 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 21:20:59.449901 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:20:59.449955 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:20:59.449975 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:20:59.471513 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:20:59.739838 3798194 disk_info.go:125] begin check local disk info of client I1121 21:20:59.743655 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:20:59.743664 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000449940 0xc000449980] E1121 21:21:06.413252 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:21:06.413276 3798194 memory.go:184] no items to output this cycle I1121 21:21:06.413435 3798194 cpu.go:275] no items to output this cycle E1121 21:21:16.412711 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:21:16.412736 3798194 memory.go:184] no items to output this cycle I1121 21:21:16.412892 3798194 cpu.go:275] no items to output this cycle I1121 21:21:17.558270 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:21:17.558279 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:21:26.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:21:26.413276 3798194 memory.go:191] Add success. I1121 21:21:26.417461 3798194 cpu.go:282] Add success. I1121 21:21:26.441949 3798194 net.go:642] Add success. I1121 21:21:26.443388 3798194 net.go:778] primary dev: ETH0 I1121 21:21:26.443403 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:21:26.443417 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:21:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:21:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:21:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:21:36.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:21:36.413259 3798194 memory.go:184] no items to output this cycle I1121 21:21:36.413406 3798194 cpu.go:275] no items to output this cycle E1121 21:21:46.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:21:46.412263 3798194 memory.go:184] no items to output this cycle I1121 21:21:46.412409 3798194 cpu.go:275] no items to output this cycle E1121 21:21:56.411908 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:21:56.411941 3798194 memory.go:191] Add success. W1121 21:21:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:21:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:21:56.411991 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:21:56.412339 3798194 cpu.go:282] Add success. I1121 21:21:56.441854 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:21:56.442380 3798194 net.go:642] Add success. I1121 21:21:56.446941 3798194 net.go:778] primary dev: ETH0 I1121 21:21:56.446957 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:21:56.446969 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:21:57.450872 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:21:57.451073 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:21:57.451084 3798194 disk_worker.go:723] disk space is not compliant W1121 21:21:57.451087 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:21:57.459387 3798194 disk_worker.go:509] system disk:vda1 I1121 21:21:57.459429 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:21:58.445994 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:21:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:21:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:21:59.450133 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:21:59.471659 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:21:59.743751 3798194 disk_info.go:125] begin check local disk info of client I1121 21:21:59.747642 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:21:59.747651 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f03c0 0xc0003f0400] E1121 21:22:06.412092 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:22:06.412132 3798194 memory.go:184] no items to output this cycle I1121 21:22:06.412279 3798194 cpu.go:275] no items to output this cycle E1121 21:22:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:22:16.413262 3798194 memory.go:184] no items to output this cycle I1121 21:22:16.413413 3798194 cpu.go:275] no items to output this cycle E1121 21:22:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:22:26.411952 3798194 memory.go:191] Add success. I1121 21:22:26.412316 3798194 cpu.go:282] Add success. I1121 21:22:26.442145 3798194 net.go:642] Add success. I1121 21:22:26.446592 3798194 net.go:778] primary dev: ETH0 I1121 21:22:26.446608 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:22:26.446628 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:22:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:22:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:22:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:22:36.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:22:36.413272 3798194 memory.go:184] no items to output this cycle I1121 21:22:36.413421 3798194 cpu.go:275] no items to output this cycle E1121 21:22:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:22:46.411973 3798194 memory.go:184] no items to output this cycle I1121 21:22:46.412125 3798194 cpu.go:275] no items to output this cycle E1121 21:22:56.413009 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:22:56.413042 3798194 memory.go:191] Add success. W1121 21:22:56.413080 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:22:56.413091 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:22:56.413095 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:22:56.417385 3798194 cpu.go:282] Add success. I1121 21:22:56.441750 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:22:56.442249 3798194 net.go:642] Add success. I1121 21:22:56.443752 3798194 net.go:778] primary dev: ETH0 I1121 21:22:56.443767 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:22:56.443779 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:22:57.454161 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:22:57.454369 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:22:57.454380 3798194 disk_worker.go:723] disk space is not compliant W1121 21:22:57.454384 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:22:57.466965 3798194 disk_worker.go:509] system disk:vda1 I1121 21:22:57.467010 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:22:58.445984 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:22:59.450083 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:22:59.450178 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:22:59.450206 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:22:59.470687 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:22:59.747750 3798194 disk_info.go:125] begin check local disk info of client I1121 21:22:59.751672 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:22:59.751682 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a4800 0xc0004a4840] E1121 21:23:06.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:23:06.413266 3798194 memory.go:184] no items to output this cycle I1121 21:23:06.413412 3798194 cpu.go:275] no items to output this cycle E1121 21:23:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:23:16.413258 3798194 memory.go:184] no items to output this cycle I1121 21:23:16.413404 3798194 cpu.go:275] no items to output this cycle E1121 21:23:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:23:26.413271 3798194 memory.go:191] Add success. I1121 21:23:26.417683 3798194 cpu.go:282] Add success. I1121 21:23:26.442279 3798194 net.go:642] Add success. I1121 21:23:26.443835 3798194 net.go:778] primary dev: ETH0 I1121 21:23:26.443849 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:23:26.443862 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:23:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:23:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:23:29.450496 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:23:36.412682 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:23:36.412704 3798194 memory.go:184] no items to output this cycle I1121 21:23:36.412845 3798194 cpu.go:275] no items to output this cycle E1121 21:23:46.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:23:46.413263 3798194 memory.go:184] no items to output this cycle I1121 21:23:46.413409 3798194 cpu.go:275] no items to output this cycle E1121 21:23:56.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:23:56.413280 3798194 memory.go:191] Add success. W1121 21:23:56.413316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:23:56.413328 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:23:56.413332 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:23:56.417883 3798194 cpu.go:282] Add success. I1121 21:23:56.441267 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:23:56.442027 3798194 net.go:642] Add success. I1121 21:23:56.444143 3798194 net.go:778] primary dev: ETH0 I1121 21:23:56.444158 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:23:56.444170 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:23:57.449264 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:23:57.449456 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:23:57.449515 3798194 disk_worker.go:723] disk space is not compliant W1121 21:23:57.449519 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:23:57.451342 3798194 disk_worker.go:509] system disk:vda1 I1121 21:23:57.451380 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:23:58.446386 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:23:59.450006 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:23:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:23:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:23:59.471674 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:23:59.751799 3798194 disk_info.go:125] begin check local disk info of client I1121 21:23:59.756979 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:23:59.756991 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004e19c0 0xc0004e1a00] E1121 21:24:06.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:24:06.411990 3798194 memory.go:184] no items to output this cycle I1121 21:24:06.412157 3798194 cpu.go:275] no items to output this cycle E1121 21:24:16.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:24:16.413272 3798194 memory.go:184] no items to output this cycle I1121 21:24:16.413427 3798194 cpu.go:275] no items to output this cycle I1121 21:24:17.558444 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:24:17.558455 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:24:26.411223 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:24:26.411257 3798194 memory.go:191] Add success. I1121 21:24:26.411620 3798194 cpu.go:282] Add success. I1121 21:24:26.442240 3798194 net.go:642] Add success. I1121 21:24:26.446507 3798194 net.go:778] primary dev: ETH0 I1121 21:24:26.446523 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:24:26.446537 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:24:29.450008 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:24:29.450081 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:24:29.450108 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:24:36.413429 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:24:36.413451 3798194 memory.go:184] no items to output this cycle I1121 21:24:36.413596 3798194 cpu.go:275] no items to output this cycle E1121 21:24:46.411973 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:24:46.411999 3798194 memory.go:184] no items to output this cycle I1121 21:24:46.412158 3798194 cpu.go:275] no items to output this cycle E1121 21:24:56.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:24:56.411960 3798194 memory.go:191] Add success. W1121 21:24:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:24:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:24:56.412008 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:24:56.412359 3798194 cpu.go:282] Add success. I1121 21:24:56.441853 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:24:56.442539 3798194 net.go:642] Add success. I1121 21:24:56.446699 3798194 net.go:778] primary dev: ETH0 I1121 21:24:56.446714 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:24:56.446729 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:24:57.450765 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:24:57.450977 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:24:57.450988 3798194 disk_worker.go:723] disk space is not compliant W1121 21:24:57.450991 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:24:57.462322 3798194 disk_worker.go:509] system disk:vda1 I1121 21:24:57.462364 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:24:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:24:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:24:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:24:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:24:59.471699 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:24:59.759945 3798194 disk_info.go:125] begin check local disk info of client I1121 21:24:59.763859 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:24:59.763869 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00042e840 0xc00042e880] E1121 21:25:06.412096 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:25:06.412137 3798194 memory.go:184] no items to output this cycle I1121 21:25:06.412285 3798194 cpu.go:275] no items to output this cycle E1121 21:25:16.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:25:16.412251 3798194 memory.go:184] no items to output this cycle I1121 21:25:16.412392 3798194 cpu.go:275] no items to output this cycle E1121 21:25:26.412870 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:25:26.412918 3798194 memory.go:191] Add success. I1121 21:25:26.417250 3798194 cpu.go:282] Add success. I1121 21:25:26.441737 3798194 net.go:642] Add success. I1121 21:25:26.443212 3798194 net.go:778] primary dev: ETH0 I1121 21:25:26.443228 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:25:26.443242 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:25:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:25:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:25:29.450149 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:25:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:25:36.411976 3798194 memory.go:184] no items to output this cycle I1121 21:25:36.412136 3798194 cpu.go:275] no items to output this cycle E1121 21:25:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:25:46.411984 3798194 memory.go:184] no items to output this cycle I1121 21:25:46.412144 3798194 cpu.go:275] no items to output this cycle E1121 21:25:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:25:56.411949 3798194 memory.go:191] Add success. W1121 21:25:56.411981 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:25:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:25:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:25:56.412358 3798194 cpu.go:282] Add success. I1121 21:25:56.441579 3798194 event_worker.go:152] Polling the log file for events... I1121 21:25:56.444704 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:25:56.445230 3798194 net.go:642] Add success. I1121 21:25:56.449938 3798194 net.go:778] primary dev: ETH0 I1121 21:25:56.449953 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:25:56.449965 3798194 net.go:659] Add eni metrics success, count: 7 E1121 21:25:57.451401 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 21:25:57.451415 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:25:57.451618 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:25:57.451627 3798194 disk_worker.go:723] disk space is not compliant W1121 21:25:57.451630 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:25:57.460006 3798194 disk_worker.go:509] system disk:vda1 I1121 21:25:57.460063 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 21:25:58.448147 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 21:25:58.448162 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:25:59.450051 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:25:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:25:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:25:59.450272 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 21:25:59.471719 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:25:59.763973 3798194 disk_info.go:125] begin check local disk info of client I1121 21:25:59.767768 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:25:59.767777 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030d580 0xc00030d5c0] E1121 21:26:06.412985 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:26:06.413011 3798194 memory.go:184] no items to output this cycle I1121 21:26:06.413190 3798194 cpu.go:275] no items to output this cycle E1121 21:26:16.412941 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:26:16.412965 3798194 memory.go:184] no items to output this cycle I1121 21:26:16.413126 3798194 cpu.go:275] no items to output this cycle E1121 21:26:26.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:26:26.411956 3798194 memory.go:191] Add success. I1121 21:26:26.412304 3798194 cpu.go:282] Add success. I1121 21:26:26.441759 3798194 net.go:642] Add success. I1121 21:26:26.446468 3798194 net.go:778] primary dev: ETH0 I1121 21:26:26.446484 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:26:26.446499 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:26:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:26:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:26:29.450125 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:26:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:26:36.412259 3798194 memory.go:184] no items to output this cycle I1121 21:26:36.412405 3798194 cpu.go:275] no items to output this cycle E1121 21:26:46.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:26:46.413269 3798194 memory.go:184] no items to output this cycle I1121 21:26:46.413421 3798194 cpu.go:275] no items to output this cycle E1121 21:26:56.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:26:56.413004 3798194 memory.go:191] Add success. W1121 21:26:56.413042 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:26:56.413055 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:26:56.413059 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:26:56.418133 3798194 cpu.go:282] Add success. I1121 21:26:56.441653 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:26:56.442133 3798194 net.go:642] Add success. I1121 21:26:56.443591 3798194 net.go:778] primary dev: ETH0 I1121 21:26:56.443605 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:26:56.443619 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:26:57.445533 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:26:57.445810 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:26:57.445822 3798194 disk_worker.go:723] disk space is not compliant W1121 21:26:57.445826 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:26:57.450784 3798194 disk_worker.go:509] system disk:vda1 I1121 21:26:57.450824 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:26:58.445983 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:26:59.450494 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:26:59.450570 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:26:59.450595 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:26:59.471089 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:26:59.768160 3798194 disk_info.go:125] begin check local disk info of client I1121 21:26:59.772059 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:26:59.772069 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ee00 0xc00030ee40] E1121 21:27:06.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:27:06.413257 3798194 memory.go:184] no items to output this cycle I1121 21:27:06.413404 3798194 cpu.go:275] no items to output this cycle E1121 21:27:16.411975 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:27:16.412001 3798194 memory.go:184] no items to output this cycle I1121 21:27:16.412162 3798194 cpu.go:275] no items to output this cycle I1121 21:27:17.558628 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:27:17.558636 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:27:26.411453 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:27:26.411487 3798194 memory.go:191] Add success. I1121 21:27:26.411839 3798194 cpu.go:282] Add success. I1121 21:27:26.441298 3798194 net.go:642] Add success. I1121 21:27:26.445784 3798194 net.go:778] primary dev: ETH0 I1121 21:27:26.445799 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:27:26.445812 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:27:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:27:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:27:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:27:36.413185 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:27:36.413208 3798194 memory.go:184] no items to output this cycle I1121 21:27:36.413346 3798194 cpu.go:275] no items to output this cycle E1121 21:27:46.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:27:46.411990 3798194 memory.go:184] no items to output this cycle I1121 21:27:46.412147 3798194 cpu.go:275] no items to output this cycle E1121 21:27:56.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:27:56.412265 3798194 memory.go:191] Add success. W1121 21:27:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:27:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:27:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:27:56.416458 3798194 cpu.go:282] Add success. I1121 21:27:56.441875 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:27:56.442391 3798194 net.go:642] Add success. I1121 21:27:56.443838 3798194 net.go:778] primary dev: ETH0 I1121 21:27:56.443852 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:27:56.443866 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:27:57.450697 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:27:57.450866 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:27:57.450937 3798194 disk_worker.go:723] disk space is not compliant W1121 21:27:57.450940 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:27:57.459922 3798194 disk_worker.go:509] system disk:vda1 I1121 21:27:57.459962 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:27:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:27:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:27:59.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:27:59.450496 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:27:59.471038 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:27:59.773135 3798194 disk_info.go:125] begin check local disk info of client I1121 21:27:59.776994 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:27:59.777003 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040bd00 0xc00040bd40] E1121 21:28:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:28:06.412259 3798194 memory.go:184] no items to output this cycle I1121 21:28:06.412708 3798194 cpu.go:275] no items to output this cycle E1121 21:28:16.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:28:16.413266 3798194 memory.go:184] no items to output this cycle I1121 21:28:16.413416 3798194 cpu.go:275] no items to output this cycle E1121 21:28:26.413084 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:28:26.413143 3798194 memory.go:191] Add success. I1121 21:28:26.417585 3798194 cpu.go:282] Add success. I1121 21:28:26.442554 3798194 net.go:642] Add success. I1121 21:28:26.448328 3798194 net.go:778] primary dev: ETH0 I1121 21:28:26.448352 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:28:26.448372 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:28:29.450573 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:28:29.450650 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:28:29.450681 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:28:36.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:28:36.413268 3798194 memory.go:184] no items to output this cycle I1121 21:28:36.413411 3798194 cpu.go:275] no items to output this cycle E1121 21:28:46.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:28:46.413259 3798194 memory.go:184] no items to output this cycle I1121 21:28:46.413425 3798194 cpu.go:275] no items to output this cycle E1121 21:28:56.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:28:56.412895 3798194 memory.go:191] Add success. W1121 21:28:56.412929 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:28:56.412942 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:28:56.412945 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:28:56.417616 3798194 cpu.go:282] Add success. I1121 21:28:56.440959 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:28:56.441553 3798194 net.go:642] Add success. I1121 21:28:56.443245 3798194 net.go:778] primary dev: ETH0 I1121 21:28:56.443259 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:28:56.443272 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:28:57.450729 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:28:57.450893 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:28:57.450951 3798194 disk_worker.go:723] disk space is not compliant W1121 21:28:57.450954 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:28:57.459321 3798194 disk_worker.go:509] system disk:vda1 I1121 21:28:57.459376 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:28:58.445983 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:28:59.450085 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:28:59.450181 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:28:59.450208 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:28:59.470694 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:28:59.777106 3798194 disk_info.go:125] begin check local disk info of client I1121 21:28:59.781014 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:28:59.781023 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000372fc0 0xc000373000] E1121 21:29:06.412805 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:29:06.412828 3798194 memory.go:184] no items to output this cycle I1121 21:29:06.412969 3798194 cpu.go:275] no items to output this cycle E1121 21:29:16.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:29:16.412265 3798194 memory.go:184] no items to output this cycle I1121 21:29:16.412416 3798194 cpu.go:275] no items to output this cycle E1121 21:29:26.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:29:26.412882 3798194 memory.go:191] Add success. I1121 21:29:26.417307 3798194 cpu.go:282] Add success. I1121 21:29:26.441746 3798194 net.go:642] Add success. I1121 21:29:26.443240 3798194 net.go:778] primary dev: ETH0 I1121 21:29:26.443256 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:29:26.443269 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:29:29.449661 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:29:29.449738 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:29:29.449763 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:29:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:29:36.412258 3798194 memory.go:184] no items to output this cycle I1121 21:29:36.412402 3798194 cpu.go:275] no items to output this cycle E1121 21:29:46.413254 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:29:46.413283 3798194 memory.go:184] no items to output this cycle I1121 21:29:46.413431 3798194 cpu.go:275] no items to output this cycle E1121 21:29:56.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:29:56.412897 3798194 memory.go:191] Add success. W1121 21:29:56.412932 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:29:56.412944 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:29:56.412947 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:29:56.417151 3798194 cpu.go:282] Add success. I1121 21:29:56.441571 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:29:56.442053 3798194 net.go:642] Add success. I1121 21:29:56.443681 3798194 net.go:778] primary dev: ETH0 I1121 21:29:56.443696 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:29:56.443709 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:29:57.451201 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:29:57.451415 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:29:57.451426 3798194 disk_worker.go:723] disk space is not compliant W1121 21:29:57.451429 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:29:57.459764 3798194 disk_worker.go:509] system disk:vda1 I1121 21:29:57.459808 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:29:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:29:59.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:29:59.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:29:59.450508 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:29:59.471024 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:29:59.781138 3798194 disk_info.go:125] begin check local disk info of client I1121 21:29:59.784975 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:29:59.784984 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f31c0 0xc0003f3200] E1121 21:30:06.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:30:06.413271 3798194 memory.go:184] no items to output this cycle I1121 21:30:06.413422 3798194 cpu.go:275] no items to output this cycle E1121 21:30:16.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:30:16.412267 3798194 memory.go:184] no items to output this cycle I1121 21:30:16.412422 3798194 cpu.go:275] no items to output this cycle I1121 21:30:17.559267 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:30:17.559275 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:30:26.412615 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:30:26.412654 3798194 memory.go:191] Add success. I1121 21:30:26.417210 3798194 cpu.go:282] Add success. I1121 21:30:26.441674 3798194 net.go:642] Add success. I1121 21:30:26.443288 3798194 net.go:778] primary dev: ETH0 I1121 21:30:26.443303 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:30:26.443318 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:30:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:30:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:30:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:30:36.413000 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:30:36.413023 3798194 memory.go:184] no items to output this cycle I1121 21:30:36.413201 3798194 cpu.go:275] no items to output this cycle E1121 21:30:46.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:30:46.411969 3798194 memory.go:184] no items to output this cycle I1121 21:30:46.412132 3798194 cpu.go:275] no items to output this cycle E1121 21:30:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:30:56.411952 3798194 memory.go:191] Add success. W1121 21:30:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:30:56.412003 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:30:56.412006 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:30:56.412357 3798194 cpu.go:282] Add success. I1121 21:30:56.445210 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:30:56.445707 3798194 net.go:642] Add success. I1121 21:30:56.450149 3798194 net.go:778] primary dev: ETH0 I1121 21:30:56.450165 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:30:56.450176 3798194 net.go:659] Add eni metrics success, count: 7 E1121 21:30:57.446461 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 21:30:57.446482 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:30:57.446731 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:30:57.446742 3798194 disk_worker.go:723] disk space is not compliant W1121 21:30:57.446745 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:30:57.449561 3798194 disk_worker.go:509] system disk:vda1 I1121 21:30:57.449602 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 21:30:58.448510 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 21:30:58.448531 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 21:30:59.448982 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 21:30:59.450056 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:30:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:30:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:30:59.471670 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:30:59.786145 3798194 disk_info.go:125] begin check local disk info of client I1121 21:30:59.790059 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:30:59.790068 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000479840 0xc000479880] E1121 21:31:06.412966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:31:06.412989 3798194 memory.go:184] no items to output this cycle I1121 21:31:06.413159 3798194 cpu.go:275] no items to output this cycle E1121 21:31:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:31:16.411986 3798194 memory.go:184] no items to output this cycle I1121 21:31:16.412145 3798194 cpu.go:275] no items to output this cycle E1121 21:31:26.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:31:26.411980 3798194 memory.go:191] Add success. I1121 21:31:26.412327 3798194 cpu.go:282] Add success. I1121 21:31:26.441739 3798194 net.go:642] Add success. I1121 21:31:26.445856 3798194 net.go:778] primary dev: ETH0 I1121 21:31:26.445873 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:31:26.445887 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:31:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:31:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:31:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:31:36.412179 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:31:36.412203 3798194 memory.go:184] no items to output this cycle I1121 21:31:36.412343 3798194 cpu.go:275] no items to output this cycle E1121 21:31:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:31:46.411978 3798194 memory.go:184] no items to output this cycle I1121 21:31:46.412132 3798194 cpu.go:275] no items to output this cycle E1121 21:31:56.413304 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:31:56.413337 3798194 memory.go:191] Add success. W1121 21:31:56.413370 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:31:56.413382 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:31:56.413386 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:31:56.418091 3798194 cpu.go:282] Add success. I1121 21:31:56.441430 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:31:56.441907 3798194 net.go:642] Add success. I1121 21:31:56.443360 3798194 net.go:778] primary dev: ETH0 I1121 21:31:56.443375 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:31:56.443387 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:31:57.446427 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:31:57.446630 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:31:57.446694 3798194 disk_worker.go:723] disk space is not compliant W1121 21:31:57.446698 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:31:57.449476 3798194 disk_worker.go:509] system disk:vda1 I1121 21:31:57.449517 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:31:58.446381 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:31:59.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:31:59.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:31:59.450489 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:31:59.471016 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:31:59.791133 3798194 disk_info.go:125] begin check local disk info of client I1121 21:31:59.794964 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:31:59.794973 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ee4c0 0xc0003ee500] E1121 21:32:06.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:32:06.412883 3798194 memory.go:184] no items to output this cycle I1121 21:32:06.413030 3798194 cpu.go:275] no items to output this cycle E1121 21:32:16.411470 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:32:16.411493 3798194 memory.go:184] no items to output this cycle I1121 21:32:16.411650 3798194 cpu.go:275] no items to output this cycle E1121 21:32:26.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:32:26.412890 3798194 memory.go:191] Add success. I1121 21:32:26.417276 3798194 cpu.go:282] Add success. I1121 21:32:26.442170 3798194 net.go:642] Add success. I1121 21:32:26.444131 3798194 net.go:778] primary dev: ETH0 I1121 21:32:26.444150 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:32:26.444174 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:32:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:32:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:32:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:32:36.415250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:32:36.415274 3798194 memory.go:184] no items to output this cycle I1121 21:32:36.415414 3798194 cpu.go:275] no items to output this cycle E1121 21:32:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:32:46.411978 3798194 memory.go:184] no items to output this cycle I1121 21:32:46.412138 3798194 cpu.go:275] no items to output this cycle E1121 21:32:56.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:32:56.413273 3798194 memory.go:191] Add success. W1121 21:32:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:32:56.413317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:32:56.413322 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:32:56.417930 3798194 cpu.go:282] Add success. I1121 21:32:56.441357 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:32:56.441972 3798194 net.go:642] Add success. I1121 21:32:56.444207 3798194 net.go:778] primary dev: ETH0 I1121 21:32:56.444223 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:32:56.444236 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:32:57.446475 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:32:57.446763 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:32:57.446775 3798194 disk_worker.go:723] disk space is not compliant W1121 21:32:57.446778 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:32:57.449628 3798194 disk_worker.go:509] system disk:vda1 I1121 21:32:57.449668 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:32:58.446370 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:32:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:32:59.450489 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:32:59.450514 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:32:59.471029 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:32:59.795143 3798194 disk_info.go:125] begin check local disk info of client I1121 21:32:59.799014 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:32:59.799023 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ee680 0xc0003ee6c0] E1121 21:33:06.411971 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:33:06.411995 3798194 memory.go:184] no items to output this cycle I1121 21:33:06.412164 3798194 cpu.go:275] no items to output this cycle E1121 21:33:16.412327 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:33:16.412352 3798194 memory.go:184] no items to output this cycle I1121 21:33:16.412504 3798194 cpu.go:275] no items to output this cycle I1121 21:33:17.559441 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:33:17.559449 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:33:26.411336 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:33:26.411369 3798194 memory.go:191] Add success. I1121 21:33:26.411691 3798194 cpu.go:282] Add success. I1121 21:33:26.441251 3798194 net.go:642] Add success. I1121 21:33:26.445396 3798194 net.go:778] primary dev: ETH0 I1121 21:33:26.445412 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:33:26.445424 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:33:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:33:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:33:29.450145 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:33:36.412345 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:33:36.412368 3798194 memory.go:184] no items to output this cycle I1121 21:33:36.412514 3798194 cpu.go:275] no items to output this cycle E1121 21:33:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:33:46.412254 3798194 memory.go:184] no items to output this cycle I1121 21:33:46.412400 3798194 cpu.go:275] no items to output this cycle E1121 21:33:56.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:33:56.413264 3798194 memory.go:191] Add success. W1121 21:33:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:33:56.413310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:33:56.413313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:33:56.417954 3798194 cpu.go:282] Add success. I1121 21:33:56.440948 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:33:56.441521 3798194 net.go:642] Add success. I1121 21:33:56.442991 3798194 net.go:778] primary dev: ETH0 I1121 21:33:56.443005 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:33:56.443017 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:33:57.453904 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:33:57.454151 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:33:57.454163 3798194 disk_worker.go:723] disk space is not compliant W1121 21:33:57.454166 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:33:57.466979 3798194 disk_worker.go:509] system disk:vda1 I1121 21:33:57.467033 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:33:58.446377 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:33:59.450483 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:33:59.450574 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:33:59.450600 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:33:59.471126 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:33:59.799166 3798194 disk_info.go:125] begin check local disk info of client I1121 21:33:59.804561 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:33:59.804575 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ee80 0xc00030eec0] E1121 21:34:06.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:34:06.412874 3798194 memory.go:184] no items to output this cycle I1121 21:34:06.413018 3798194 cpu.go:275] no items to output this cycle E1121 21:34:16.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:34:16.412255 3798194 memory.go:184] no items to output this cycle I1121 21:34:16.412402 3798194 cpu.go:275] no items to output this cycle E1121 21:34:26.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:34:26.412890 3798194 memory.go:191] Add success. I1121 21:34:26.418016 3798194 cpu.go:282] Add success. I1121 21:34:26.441583 3798194 net.go:642] Add success. I1121 21:34:26.443056 3798194 net.go:778] primary dev: ETH0 I1121 21:34:26.443071 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:34:26.443083 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:34:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:34:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:34:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:34:36.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:34:36.412249 3798194 memory.go:184] no items to output this cycle I1121 21:34:36.412388 3798194 cpu.go:275] no items to output this cycle E1121 21:34:46.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:34:46.412883 3798194 memory.go:184] no items to output this cycle I1121 21:34:46.413029 3798194 cpu.go:275] no items to output this cycle E1121 21:34:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:34:56.411959 3798194 memory.go:191] Add success. W1121 21:34:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:34:56.412005 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:34:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:34:56.412360 3798194 cpu.go:282] Add success. I1121 21:34:56.441811 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:34:56.442334 3798194 net.go:642] Add success. I1121 21:34:56.446592 3798194 net.go:778] primary dev: ETH0 I1121 21:34:56.446607 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:34:56.446621 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:34:57.450612 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:34:57.450817 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:34:57.450827 3798194 disk_worker.go:723] disk space is not compliant W1121 21:34:57.450830 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:34:57.458971 3798194 disk_worker.go:509] system disk:vda1 I1121 21:34:57.459012 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:34:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:34:59.450486 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:34:59.450569 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:34:59.450598 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:34:59.471132 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:34:59.810151 3798194 disk_info.go:125] begin check local disk info of client I1121 21:34:59.827449 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:34:59.827458 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00044e1c0 0xc00044e200] E1121 21:35:06.412098 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:35:06.412138 3798194 memory.go:184] no items to output this cycle I1121 21:35:06.412284 3798194 cpu.go:275] no items to output this cycle E1121 21:35:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:35:16.412257 3798194 memory.go:184] no items to output this cycle I1121 21:35:16.412413 3798194 cpu.go:275] no items to output this cycle E1121 21:35:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:35:26.413268 3798194 memory.go:191] Add success. I1121 21:35:26.418172 3798194 cpu.go:282] Add success. I1121 21:35:26.441741 3798194 net.go:642] Add success. I1121 21:35:26.443355 3798194 net.go:778] primary dev: ETH0 I1121 21:35:26.443370 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:35:26.443383 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:35:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:35:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:35:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:35:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:35:36.412256 3798194 memory.go:184] no items to output this cycle I1121 21:35:36.412400 3798194 cpu.go:275] no items to output this cycle E1121 21:35:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:35:46.411976 3798194 memory.go:184] no items to output this cycle I1121 21:35:46.412133 3798194 cpu.go:275] no items to output this cycle E1121 21:35:56.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:35:56.411955 3798194 memory.go:191] Add success. W1121 21:35:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:35:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:35:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:35:56.412345 3798194 cpu.go:282] Add success. I1121 21:35:56.440887 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:35:56.442142 3798194 event_worker.go:152] Polling the log file for events... I1121 21:35:56.442883 3798194 net.go:642] Add success. I1121 21:35:56.448775 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795266356, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 21:35:56.448886 3798194 net.go:778] primary dev: ETH0 I1121 21:35:56.448901 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:35:56.448921 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:35:56.465324 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 21:35:57.445304 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 21:35:57.445323 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:35:57.445582 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:35:57.445593 3798194 disk_worker.go:723] disk space is not compliant W1121 21:35:57.445596 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:35:57.448103 3798194 disk_worker.go:509] system disk:vda1 I1121 21:35:57.448153 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 21:35:58.448144 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 21:35:58.448159 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 21:35:59.448635 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 21:35:59.449720 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:35:59.449783 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:35:59.449803 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:35:59.471333 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:35:59.827555 3798194 disk_info.go:125] begin check local disk info of client I1121 21:35:59.831431 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:35:59.831440 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037c500 0xc00037c540] E1121 21:36:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:36:06.411976 3798194 memory.go:184] no items to output this cycle I1121 21:36:06.412129 3798194 cpu.go:275] no items to output this cycle E1121 21:36:16.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:36:16.412885 3798194 memory.go:184] no items to output this cycle I1121 21:36:16.413028 3798194 cpu.go:275] no items to output this cycle I1121 21:36:17.560270 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:36:17.560278 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:36:26.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:36:26.412881 3798194 memory.go:191] Add success. I1121 21:36:26.417747 3798194 cpu.go:282] Add success. I1121 21:36:26.442280 3798194 net.go:642] Add success. I1121 21:36:26.443787 3798194 net.go:778] primary dev: ETH0 I1121 21:36:26.443803 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:36:26.443815 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:36:29.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:36:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:36:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:36:36.413302 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:36:36.413333 3798194 memory.go:184] no items to output this cycle I1121 21:36:36.413555 3798194 cpu.go:275] no items to output this cycle E1121 21:36:46.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:36:46.412255 3798194 memory.go:184] no items to output this cycle I1121 21:36:46.412396 3798194 cpu.go:275] no items to output this cycle E1121 21:36:56.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:36:56.412268 3798194 memory.go:191] Add success. W1121 21:36:56.412295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:36:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:36:56.412312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:36:56.416737 3798194 cpu.go:282] Add success. I1121 21:36:56.441270 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:36:56.441752 3798194 net.go:642] Add success. I1121 21:36:56.443293 3798194 net.go:778] primary dev: ETH0 I1121 21:36:56.443307 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:36:56.443321 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:36:57.451019 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:36:57.451205 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:36:57.451260 3798194 disk_worker.go:723] disk space is not compliant W1121 21:36:57.451263 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:36:57.459980 3798194 disk_worker.go:509] system disk:vda1 I1121 21:36:57.460027 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:36:58.446521 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:36:59.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:36:59.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:36:59.450496 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:36:59.471004 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:36:59.832141 3798194 disk_info.go:125] begin check local disk info of client I1121 21:36:59.835963 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:36:59.835972 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030a8c0 0xc00030a900] E1121 21:37:06.411977 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:37:06.411998 3798194 memory.go:184] no items to output this cycle I1121 21:37:06.412148 3798194 cpu.go:275] no items to output this cycle E1121 21:37:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:37:16.411981 3798194 memory.go:184] no items to output this cycle I1121 21:37:16.412145 3798194 cpu.go:275] no items to output this cycle E1121 21:37:26.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:37:26.412268 3798194 memory.go:191] Add success. I1121 21:37:26.416368 3798194 cpu.go:282] Add success. I1121 21:37:26.442028 3798194 net.go:642] Add success. I1121 21:37:26.443578 3798194 net.go:778] primary dev: ETH0 I1121 21:37:26.443593 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:37:26.443608 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:37:29.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:37:29.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:37:29.450511 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:37:36.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:37:36.412260 3798194 memory.go:184] no items to output this cycle I1121 21:37:36.412399 3798194 cpu.go:275] no items to output this cycle E1121 21:37:46.412870 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:37:46.412893 3798194 memory.go:184] no items to output this cycle I1121 21:37:46.413039 3798194 cpu.go:275] no items to output this cycle E1121 21:37:56.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:37:56.413275 3798194 memory.go:191] Add success. W1121 21:37:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:37:56.413322 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:37:56.413326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:37:56.417456 3798194 cpu.go:282] Add success. I1121 21:37:56.441869 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:37:56.442440 3798194 net.go:642] Add success. I1121 21:37:56.443761 3798194 net.go:778] primary dev: ETH0 I1121 21:37:56.443776 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:37:56.443798 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:37:57.452213 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:37:57.452418 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:37:57.452430 3798194 disk_worker.go:723] disk space is not compliant W1121 21:37:57.452433 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:37:57.464318 3798194 disk_worker.go:509] system disk:vda1 I1121 21:37:57.464366 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:37:58.446392 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:37:59.450486 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:37:59.450571 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:37:59.450597 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:37:59.471128 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:37:59.836155 3798194 disk_info.go:125] begin check local disk info of client I1121 21:37:59.857518 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:37:59.857528 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0200 0xc0002e0240] E1121 21:38:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:38:06.411984 3798194 memory.go:184] no items to output this cycle I1121 21:38:06.412133 3798194 cpu.go:275] no items to output this cycle E1121 21:38:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:38:16.411977 3798194 memory.go:184] no items to output this cycle I1121 21:38:16.412140 3798194 cpu.go:275] no items to output this cycle E1121 21:38:26.411233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:38:26.411268 3798194 memory.go:191] Add success. I1121 21:38:26.415297 3798194 cpu.go:282] Add success. I1121 21:38:26.441742 3798194 net.go:642] Add success. I1121 21:38:26.443175 3798194 net.go:778] primary dev: ETH0 I1121 21:38:26.443189 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:38:26.443202 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:38:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:38:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:38:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:38:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:38:36.411981 3798194 memory.go:184] no items to output this cycle I1121 21:38:36.412134 3798194 cpu.go:275] no items to output this cycle E1121 21:38:46.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:38:46.411992 3798194 memory.go:184] no items to output this cycle I1121 21:38:46.412156 3798194 cpu.go:275] no items to output this cycle E1121 21:38:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:38:56.411950 3798194 memory.go:191] Add success. W1121 21:38:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:38:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:38:56.412000 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:38:56.412351 3798194 cpu.go:282] Add success. I1121 21:38:56.441805 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:38:56.442416 3798194 net.go:642] Add success. I1121 21:38:56.446649 3798194 net.go:778] primary dev: ETH0 I1121 21:38:56.446664 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:38:56.446677 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:38:57.450323 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:38:57.450491 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:38:57.450545 3798194 disk_worker.go:723] disk space is not compliant W1121 21:38:57.450548 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:38:57.458720 3798194 disk_worker.go:509] system disk:vda1 I1121 21:38:57.458778 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:38:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:38:59.450105 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:38:59.450195 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:38:59.450222 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:38:59.470729 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:38:59.858162 3798194 disk_info.go:125] begin check local disk info of client I1121 21:38:59.875535 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:38:59.875545 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004085c0 0xc000408600] E1121 21:39:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:39:06.412256 3798194 memory.go:184] no items to output this cycle I1121 21:39:06.412400 3798194 cpu.go:275] no items to output this cycle E1121 21:39:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:39:16.411972 3798194 memory.go:184] no items to output this cycle I1121 21:39:16.412140 3798194 cpu.go:275] no items to output this cycle I1121 21:39:17.560447 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:39:17.560456 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:39:26.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:39:26.412266 3798194 memory.go:191] Add success. I1121 21:39:26.416348 3798194 cpu.go:282] Add success. I1121 21:39:26.441873 3798194 net.go:642] Add success. I1121 21:39:26.443417 3798194 net.go:778] primary dev: ETH0 I1121 21:39:26.443440 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:39:26.443455 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:39:29.450406 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:39:29.450489 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:39:29.450513 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:39:36.412602 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:39:36.412627 3798194 memory.go:184] no items to output this cycle I1121 21:39:36.412784 3798194 cpu.go:275] no items to output this cycle E1121 21:39:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:39:46.411980 3798194 memory.go:184] no items to output this cycle I1121 21:39:46.412142 3798194 cpu.go:275] no items to output this cycle E1121 21:39:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:39:56.412266 3798194 memory.go:191] Add success. W1121 21:39:56.412295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:39:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:39:56.412311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:39:56.416609 3798194 cpu.go:282] Add success. I1121 21:39:56.440944 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:39:56.441524 3798194 net.go:642] Add success. I1121 21:39:56.443002 3798194 net.go:778] primary dev: ETH0 I1121 21:39:56.443017 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:39:56.443029 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:39:57.450600 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:39:57.450759 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:39:57.450811 3798194 disk_worker.go:723] disk space is not compliant W1121 21:39:57.450815 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:39:57.459200 3798194 disk_worker.go:509] system disk:vda1 I1121 21:39:57.459242 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:39:58.446368 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:39:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:39:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:39:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:39:59.471667 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:39:59.880697 3798194 disk_info.go:125] begin check local disk info of client I1121 21:39:59.884224 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:39:59.884232 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b2280 0xc0003b22c0] E1121 21:40:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:40:06.411978 3798194 memory.go:184] no items to output this cycle I1121 21:40:06.412145 3798194 cpu.go:275] no items to output this cycle E1121 21:40:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:40:16.411981 3798194 memory.go:184] no items to output this cycle I1121 21:40:16.412144 3798194 cpu.go:275] no items to output this cycle E1121 21:40:26.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:40:26.412893 3798194 memory.go:191] Add success. I1121 21:40:26.417006 3798194 cpu.go:282] Add success. I1121 21:40:26.441642 3798194 net.go:642] Add success. I1121 21:40:26.443443 3798194 net.go:778] primary dev: ETH0 I1121 21:40:26.443459 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:40:26.443488 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:40:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:40:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:40:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:40:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:40:36.411980 3798194 memory.go:184] no items to output this cycle I1121 21:40:36.412140 3798194 cpu.go:275] no items to output this cycle E1121 21:40:46.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:40:46.412268 3798194 memory.go:184] no items to output this cycle I1121 21:40:46.412416 3798194 cpu.go:275] no items to output this cycle E1121 21:40:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:40:56.411943 3798194 memory.go:191] Add success. W1121 21:40:56.411971 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:40:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:40:56.411986 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:40:56.412346 3798194 cpu.go:282] Add success. I1121 21:40:56.444485 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:40:56.444956 3798194 net.go:642] Add success. I1121 21:40:56.449164 3798194 net.go:778] primary dev: ETH0 I1121 21:40:56.449179 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:40:56.449193 3798194 net.go:659] Add eni metrics success, count: 7 E1121 21:40:57.450940 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 21:40:57.450954 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:40:57.451134 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:40:57.451202 3798194 disk_worker.go:723] disk space is not compliant W1121 21:40:57.451205 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:40:57.460067 3798194 disk_worker.go:509] system disk:vda1 I1121 21:40:57.460126 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 21:40:58.448703 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 21:40:58.448718 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 21:40:59.448262 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 21:40:59.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:40:59.450442 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:40:59.450464 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:40:59.470978 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:40:59.885147 3798194 disk_info.go:125] begin check local disk info of client I1121 21:40:59.889006 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:40:59.889014 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00034bdc0 0xc00034be00] E1121 21:41:06.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:41:06.412252 3798194 memory.go:184] no items to output this cycle I1121 21:41:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 21:41:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:41:16.412263 3798194 memory.go:184] no items to output this cycle I1121 21:41:16.412411 3798194 cpu.go:275] no items to output this cycle E1121 21:41:26.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:41:26.412269 3798194 memory.go:191] Add success. I1121 21:41:26.416831 3798194 cpu.go:282] Add success. I1121 21:41:26.441286 3798194 net.go:642] Add success. I1121 21:41:26.442851 3798194 net.go:778] primary dev: ETH0 I1121 21:41:26.442867 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:41:26.442879 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:41:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:41:29.450088 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:41:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:41:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:41:36.411980 3798194 memory.go:184] no items to output this cycle I1121 21:41:36.412133 3798194 cpu.go:275] no items to output this cycle E1121 21:41:46.412815 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:41:46.412837 3798194 memory.go:184] no items to output this cycle I1121 21:41:46.412983 3798194 cpu.go:275] no items to output this cycle E1121 21:41:56.411910 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:41:56.411944 3798194 memory.go:191] Add success. W1121 21:41:56.411972 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:41:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:41:56.411987 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:41:56.412343 3798194 cpu.go:282] Add success. I1121 21:41:56.441802 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:41:56.442832 3798194 net.go:642] Add success. I1121 21:41:56.447046 3798194 net.go:778] primary dev: ETH0 I1121 21:41:56.447068 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:41:56.447188 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:41:57.450643 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:41:57.450798 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:41:57.450850 3798194 disk_worker.go:723] disk space is not compliant W1121 21:41:57.450853 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:41:57.459214 3798194 disk_worker.go:509] system disk:vda1 I1121 21:41:57.459256 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:41:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:41:59.450104 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:41:59.450192 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:41:59.450216 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:41:59.470685 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:41:59.890152 3798194 disk_info.go:125] begin check local disk info of client I1121 21:41:59.893999 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:41:59.894007 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004d9bc0 0xc0004d9c00] E1121 21:42:06.412089 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:42:06.412128 3798194 memory.go:184] no items to output this cycle I1121 21:42:06.412276 3798194 cpu.go:275] no items to output this cycle E1121 21:42:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:42:16.411973 3798194 memory.go:184] no items to output this cycle I1121 21:42:16.412133 3798194 cpu.go:275] no items to output this cycle I1121 21:42:17.560616 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:42:17.560624 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:42:26.411555 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:42:26.411589 3798194 memory.go:191] Add success. I1121 21:42:26.411949 3798194 cpu.go:282] Add success. I1121 21:42:26.441570 3798194 net.go:642] Add success. I1121 21:42:26.445747 3798194 net.go:778] primary dev: ETH0 I1121 21:42:26.445764 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:42:26.445777 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:42:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:42:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:42:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:42:36.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:42:36.413265 3798194 memory.go:184] no items to output this cycle I1121 21:42:36.413413 3798194 cpu.go:275] no items to output this cycle E1121 21:42:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:42:46.411974 3798194 memory.go:184] no items to output this cycle I1121 21:42:46.412130 3798194 cpu.go:275] no items to output this cycle E1121 21:42:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:42:56.411955 3798194 memory.go:191] Add success. W1121 21:42:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:42:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:42:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:42:56.412346 3798194 cpu.go:282] Add success. I1121 21:42:56.441810 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:42:56.442407 3798194 net.go:642] Add success. I1121 21:42:56.447243 3798194 net.go:778] primary dev: ETH0 I1121 21:42:56.447259 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:42:56.447271 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:42:57.450191 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:42:57.450386 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:42:57.450396 3798194 disk_worker.go:723] disk space is not compliant W1121 21:42:57.450399 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:42:57.458465 3798194 disk_worker.go:509] system disk:vda1 I1121 21:42:57.458506 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:42:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:42:59.449950 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:42:59.450035 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:42:59.450060 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:42:59.471585 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:42:59.895142 3798194 disk_info.go:125] begin check local disk info of client I1121 21:42:59.898660 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:42:59.898669 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fe8c0 0xc0003fe900] E1121 21:43:06.413158 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:43:06.413179 3798194 memory.go:184] no items to output this cycle I1121 21:43:06.413319 3798194 cpu.go:275] no items to output this cycle E1121 21:43:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:43:16.411980 3798194 memory.go:184] no items to output this cycle I1121 21:43:16.412407 3798194 cpu.go:275] no items to output this cycle E1121 21:43:26.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:43:26.412273 3798194 memory.go:191] Add success. I1121 21:43:26.416718 3798194 cpu.go:282] Add success. I1121 21:43:26.441460 3798194 net.go:642] Add success. I1121 21:43:26.442872 3798194 net.go:778] primary dev: ETH0 I1121 21:43:26.442886 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:43:26.442899 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:43:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:43:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:43:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:43:36.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:43:36.411974 3798194 memory.go:184] no items to output this cycle I1121 21:43:36.412126 3798194 cpu.go:275] no items to output this cycle E1121 21:43:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:43:46.411989 3798194 memory.go:184] no items to output this cycle I1121 21:43:46.412147 3798194 cpu.go:275] no items to output this cycle E1121 21:43:56.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:43:56.412262 3798194 memory.go:191] Add success. W1121 21:43:56.412291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:43:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:43:56.412305 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:43:56.416883 3798194 cpu.go:282] Add success. I1121 21:43:56.441432 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:43:56.441982 3798194 net.go:642] Add success. I1121 21:43:56.443761 3798194 net.go:778] primary dev: ETH0 I1121 21:43:56.443775 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:43:56.443788 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:43:57.450901 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:43:57.451095 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:43:57.451106 3798194 disk_worker.go:723] disk space is not compliant W1121 21:43:57.451141 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:43:57.460200 3798194 disk_worker.go:509] system disk:vda1 I1121 21:43:57.460242 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:43:58.446360 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:43:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:43:59.450129 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:43:59.450156 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:43:59.471671 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:43:59.899267 3798194 disk_info.go:125] begin check local disk info of client I1121 21:43:59.918296 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:43:59.918308 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030de40 0xc00030de80] E1121 21:44:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:44:06.411977 3798194 memory.go:184] no items to output this cycle I1121 21:44:06.412135 3798194 cpu.go:275] no items to output this cycle E1121 21:44:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:44:16.411982 3798194 memory.go:184] no items to output this cycle I1121 21:44:16.412136 3798194 cpu.go:275] no items to output this cycle E1121 21:44:26.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:44:26.413261 3798194 memory.go:191] Add success. I1121 21:44:26.417691 3798194 cpu.go:282] Add success. I1121 21:44:26.441333 3798194 net.go:642] Add success. I1121 21:44:26.442821 3798194 net.go:778] primary dev: ETH0 I1121 21:44:26.442836 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:44:26.442850 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:44:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:44:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:44:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:44:36.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:44:36.411972 3798194 memory.go:184] no items to output this cycle I1121 21:44:36.412130 3798194 cpu.go:275] no items to output this cycle E1121 21:44:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:44:46.411980 3798194 memory.go:184] no items to output this cycle I1121 21:44:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 21:44:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:44:56.412273 3798194 memory.go:191] Add success. W1121 21:44:56.412301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:44:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:44:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:44:56.416640 3798194 cpu.go:282] Add success. I1121 21:44:56.440955 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:44:56.441546 3798194 net.go:642] Add success. I1121 21:44:56.443034 3798194 net.go:778] primary dev: ETH0 I1121 21:44:56.443049 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:44:56.443062 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:44:57.450634 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:44:57.450806 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:44:57.450861 3798194 disk_worker.go:723] disk space is not compliant W1121 21:44:57.450864 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:44:57.459032 3798194 disk_worker.go:509] system disk:vda1 I1121 21:44:57.459075 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:44:58.446383 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:44:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:44:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:44:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:44:59.471651 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:44:59.919145 3798194 disk_info.go:125] begin check local disk info of client I1121 21:44:59.922667 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:44:59.922678 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e1780 0xc0002e17c0] E1121 21:45:06.411975 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:45:06.411998 3798194 memory.go:184] no items to output this cycle I1121 21:45:06.412163 3798194 cpu.go:275] no items to output this cycle E1121 21:45:16.412345 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:45:16.412367 3798194 memory.go:184] no items to output this cycle I1121 21:45:16.412516 3798194 cpu.go:275] no items to output this cycle I1121 21:45:17.560785 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:45:17.560793 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:45:26.411390 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:45:26.411423 3798194 memory.go:191] Add success. I1121 21:45:26.411748 3798194 cpu.go:282] Add success. I1121 21:45:26.441305 3798194 net.go:642] Add success. I1121 21:45:26.445688 3798194 net.go:778] primary dev: ETH0 I1121 21:45:26.445703 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:45:26.445714 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:45:29.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:45:29.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:45:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:45:36.412372 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:45:36.412398 3798194 memory.go:184] no items to output this cycle I1121 21:45:36.412541 3798194 cpu.go:275] no items to output this cycle E1121 21:45:46.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:45:46.413267 3798194 memory.go:184] no items to output this cycle I1121 21:45:46.413413 3798194 cpu.go:275] no items to output this cycle E1121 21:45:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:45:56.411945 3798194 memory.go:191] Add success. W1121 21:45:56.411971 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:45:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:45:56.411985 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:45:56.412336 3798194 cpu.go:282] Add success. I1121 21:45:56.441562 3798194 event_worker.go:152] Polling the log file for events... I1121 21:45:56.444681 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:45:56.445182 3798194 net.go:642] Add success. I1121 21:45:56.449819 3798194 net.go:778] primary dev: ETH0 I1121 21:45:56.449833 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:45:56.449847 3798194 net.go:659] Add eni metrics success, count: 7 E1121 21:45:57.450557 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 21:45:57.450571 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:45:57.450770 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:45:57.450778 3798194 disk_worker.go:723] disk space is not compliant W1121 21:45:57.450782 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:45:57.459048 3798194 disk_worker.go:509] system disk:vda1 I1121 21:45:57.459088 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 21:45:58.448204 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 21:45:58.448219 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 21:45:59.448658 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 21:45:59.449733 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:45:59.449785 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:45:59.449813 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:45:59.471338 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:45:59.924145 3798194 disk_info.go:125] begin check local disk info of client I1121 21:45:59.927966 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:45:59.927974 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d82c0 0xc0003d8300] E1121 21:46:06.412760 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:46:06.412783 3798194 memory.go:184] no items to output this cycle I1121 21:46:06.412945 3798194 cpu.go:275] no items to output this cycle E1121 21:46:16.412339 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:46:16.412364 3798194 memory.go:184] no items to output this cycle I1121 21:46:16.412507 3798194 cpu.go:275] no items to output this cycle E1121 21:46:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:46:26.411955 3798194 memory.go:191] Add success. I1121 21:46:26.412309 3798194 cpu.go:282] Add success. I1121 21:46:26.441931 3798194 net.go:642] Add success. I1121 21:46:26.446258 3798194 net.go:778] primary dev: ETH0 I1121 21:46:26.446273 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:46:26.446286 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:46:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:46:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:46:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:46:36.412775 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:46:36.412798 3798194 memory.go:184] no items to output this cycle I1121 21:46:36.412948 3798194 cpu.go:275] no items to output this cycle E1121 21:46:46.412405 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:46:46.412429 3798194 memory.go:184] no items to output this cycle I1121 21:46:46.412586 3798194 cpu.go:275] no items to output this cycle E1121 21:46:56.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:46:56.412904 3798194 memory.go:191] Add success. W1121 21:46:56.412934 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:46:56.412949 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:46:56.412954 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:46:56.417574 3798194 cpu.go:282] Add success. I1121 21:46:56.441193 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:46:56.441670 3798194 net.go:642] Add success. I1121 21:46:56.443235 3798194 net.go:778] primary dev: ETH0 I1121 21:46:56.443249 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:46:56.443262 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:46:57.450693 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:46:57.450889 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:46:57.450901 3798194 disk_worker.go:723] disk space is not compliant W1121 21:46:57.450903 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:46:57.459184 3798194 disk_worker.go:509] system disk:vda1 I1121 21:46:57.459230 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:46:58.446011 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:46:59.450123 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:46:59.450202 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:46:59.450229 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:46:59.470722 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:46:59.929174 3798194 disk_info.go:125] begin check local disk info of client I1121 21:46:59.933016 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:46:59.933025 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003af900 0xc0003af940] E1121 21:47:06.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:47:06.412882 3798194 memory.go:184] no items to output this cycle I1121 21:47:06.413029 3798194 cpu.go:275] no items to output this cycle E1121 21:47:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:47:16.412261 3798194 memory.go:184] no items to output this cycle I1121 21:47:16.412403 3798194 cpu.go:275] no items to output this cycle E1121 21:47:26.411910 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:47:26.411943 3798194 memory.go:191] Add success. I1121 21:47:26.412285 3798194 cpu.go:282] Add success. I1121 21:47:26.442002 3798194 net.go:642] Add success. I1121 21:47:26.452876 3798194 net.go:778] primary dev: ETH0 I1121 21:47:26.452892 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:47:26.452907 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:47:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:47:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:47:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:47:36.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:47:36.412250 3798194 memory.go:184] no items to output this cycle I1121 21:47:36.412393 3798194 cpu.go:275] no items to output this cycle E1121 21:47:46.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:47:46.412266 3798194 memory.go:184] no items to output this cycle I1121 21:47:46.412412 3798194 cpu.go:275] no items to output this cycle E1121 21:47:56.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:47:56.412890 3798194 memory.go:191] Add success. W1121 21:47:56.412917 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:47:56.412928 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:47:56.412931 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:47:56.417603 3798194 cpu.go:282] Add success. I1121 21:47:56.440932 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:47:56.441489 3798194 net.go:642] Add success. I1121 21:47:56.442930 3798194 net.go:778] primary dev: ETH0 I1121 21:47:56.442945 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:47:56.442959 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:47:57.451864 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:47:57.452065 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:47:57.452076 3798194 disk_worker.go:723] disk space is not compliant W1121 21:47:57.452079 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:47:57.460238 3798194 disk_worker.go:509] system disk:vda1 I1121 21:47:57.460281 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:47:58.446189 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:47:59.450407 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:47:59.450493 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:47:59.450522 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:47:59.471072 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:47:59.933136 3798194 disk_info.go:125] begin check local disk info of client I1121 21:47:59.936720 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:47:59.936729 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000426f00 0xc000426f40] E1121 21:48:06.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:48:06.412251 3798194 memory.go:184] no items to output this cycle I1121 21:48:06.412395 3798194 cpu.go:275] no items to output this cycle E1121 21:48:16.412522 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:48:16.412544 3798194 memory.go:184] no items to output this cycle I1121 21:48:16.412689 3798194 cpu.go:275] no items to output this cycle I1121 21:48:17.560947 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:48:17.560956 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:48:26.413082 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:48:26.413137 3798194 memory.go:191] Add success. I1121 21:48:26.417381 3798194 cpu.go:282] Add success. I1121 21:48:26.441889 3798194 net.go:642] Add success. I1121 21:48:26.443307 3798194 net.go:778] primary dev: ETH0 I1121 21:48:26.443322 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:48:26.443335 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:48:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:48:29.450466 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:48:29.450494 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:48:36.412841 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:48:36.412864 3798194 memory.go:184] no items to output this cycle I1121 21:48:36.413008 3798194 cpu.go:275] no items to output this cycle E1121 21:48:46.411976 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:48:46.411998 3798194 memory.go:184] no items to output this cycle I1121 21:48:46.412157 3798194 cpu.go:275] no items to output this cycle E1121 21:48:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:48:56.411952 3798194 memory.go:191] Add success. W1121 21:48:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:48:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:48:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:48:56.412365 3798194 cpu.go:282] Add success. I1121 21:48:56.441814 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:48:56.442418 3798194 net.go:642] Add success. I1121 21:48:56.446944 3798194 net.go:778] primary dev: ETH0 I1121 21:48:56.446959 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:48:56.446973 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:48:57.451030 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:48:57.451530 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:48:57.451541 3798194 disk_worker.go:723] disk space is not compliant W1121 21:48:57.451544 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:48:57.461463 3798194 disk_worker.go:509] system disk:vda1 I1121 21:48:57.461506 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:48:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:48:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:48:59.450122 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:48:59.450148 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:48:59.471671 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:48:59.936832 3798194 disk_info.go:125] begin check local disk info of client I1121 21:48:59.940489 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:48:59.940498 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a39c0 0xc0004a3a00] E1121 21:49:06.412370 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:49:06.412392 3798194 memory.go:184] no items to output this cycle I1121 21:49:06.412536 3798194 cpu.go:275] no items to output this cycle E1121 21:49:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:49:16.411982 3798194 memory.go:184] no items to output this cycle I1121 21:49:16.412152 3798194 cpu.go:275] no items to output this cycle E1121 21:49:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:49:26.412269 3798194 memory.go:191] Add success. I1121 21:49:26.425127 3798194 cpu.go:282] Add success. I1121 21:49:26.441880 3798194 net.go:642] Add success. I1121 21:49:26.443550 3798194 net.go:778] primary dev: ETH0 I1121 21:49:26.443566 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:49:26.443579 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:49:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:49:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:49:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:49:36.413224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:49:36.413249 3798194 memory.go:184] no items to output this cycle I1121 21:49:36.413393 3798194 cpu.go:275] no items to output this cycle E1121 21:49:46.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:49:46.412252 3798194 memory.go:184] no items to output this cycle I1121 21:49:46.412404 3798194 cpu.go:275] no items to output this cycle E1121 21:49:56.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:49:56.412883 3798194 memory.go:191] Add success. W1121 21:49:56.412910 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:49:56.412922 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:49:56.412925 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:49:56.417060 3798194 cpu.go:282] Add success. I1121 21:49:56.441605 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:49:56.442090 3798194 net.go:642] Add success. I1121 21:49:56.443512 3798194 net.go:778] primary dev: ETH0 I1121 21:49:56.443527 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:49:56.443539 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:49:57.450810 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:49:57.450970 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:49:57.451024 3798194 disk_worker.go:723] disk space is not compliant W1121 21:49:57.451027 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:49:57.459866 3798194 disk_worker.go:509] system disk:vda1 I1121 21:49:57.459931 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:49:58.446504 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:49:59.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:49:59.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:49:59.450507 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:49:59.471035 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:49:59.940602 3798194 disk_info.go:125] begin check local disk info of client I1121 21:49:59.944173 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:49:59.944183 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d8c40 0xc0003d8c80] E1121 21:50:06.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:50:06.411976 3798194 memory.go:184] no items to output this cycle I1121 21:50:06.412130 3798194 cpu.go:275] no items to output this cycle E1121 21:50:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:50:16.411979 3798194 memory.go:184] no items to output this cycle I1121 21:50:16.412132 3798194 cpu.go:275] no items to output this cycle E1121 21:50:26.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:50:26.412272 3798194 memory.go:191] Add success. I1121 21:50:26.416516 3798194 cpu.go:282] Add success. I1121 21:50:26.442072 3798194 net.go:642] Add success. I1121 21:50:26.443419 3798194 net.go:778] primary dev: ETH0 I1121 21:50:26.443434 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:50:26.443447 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:50:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:50:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:50:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:50:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:50:36.411980 3798194 memory.go:184] no items to output this cycle I1121 21:50:36.412139 3798194 cpu.go:275] no items to output this cycle E1121 21:50:46.412870 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:50:46.412891 3798194 memory.go:184] no items to output this cycle I1121 21:50:46.413040 3798194 cpu.go:275] no items to output this cycle E1121 21:50:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:50:56.411956 3798194 memory.go:191] Add success. W1121 21:50:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:50:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:50:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:50:56.412356 3798194 cpu.go:282] Add success. I1121 21:50:56.444792 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:50:56.445315 3798194 net.go:642] Add success. I1121 21:50:56.449213 3798194 net.go:778] primary dev: ETH0 I1121 21:50:56.449227 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:50:56.449239 3798194 net.go:659] Add eni metrics success, count: 7 E1121 21:50:57.450513 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 21:50:57.450527 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:50:57.450720 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:50:57.450729 3798194 disk_worker.go:723] disk space is not compliant W1121 21:50:57.450732 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:50:57.459099 3798194 disk_worker.go:509] system disk:vda1 I1121 21:50:57.459150 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 21:50:58.448125 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 21:50:58.448141 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 21:50:59.448605 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 21:50:59.449681 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:50:59.449733 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:50:59.449755 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:50:59.471286 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:50:59.945155 3798194 disk_info.go:125] begin check local disk info of client I1121 21:50:59.966412 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:50:59.966421 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004183c0 0xc000418400] E1121 21:51:06.412338 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:51:06.412366 3798194 memory.go:184] no items to output this cycle I1121 21:51:06.412653 3798194 cpu.go:275] no items to output this cycle E1121 21:51:16.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:51:16.411986 3798194 memory.go:184] no items to output this cycle I1121 21:51:16.412152 3798194 cpu.go:275] no items to output this cycle I1121 21:51:17.562044 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:51:17.562052 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:51:26.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:51:26.413264 3798194 memory.go:191] Add success. I1121 21:51:26.417565 3798194 cpu.go:282] Add success. I1121 21:51:26.442211 3798194 net.go:642] Add success. I1121 21:51:26.443743 3798194 net.go:778] primary dev: ETH0 I1121 21:51:26.443758 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:51:26.443772 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:51:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:51:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:51:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:51:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:51:36.412256 3798194 memory.go:184] no items to output this cycle I1121 21:51:36.412399 3798194 cpu.go:275] no items to output this cycle E1121 21:51:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:51:46.411982 3798194 memory.go:184] no items to output this cycle I1121 21:51:46.412148 3798194 cpu.go:275] no items to output this cycle E1121 21:51:56.411939 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:51:56.411976 3798194 memory.go:191] Add success. W1121 21:51:56.412013 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:51:56.412025 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:51:56.412028 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:51:56.412414 3798194 cpu.go:282] Add success. I1121 21:51:56.440973 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:51:56.441580 3798194 net.go:642] Add success. I1121 21:51:56.446348 3798194 net.go:778] primary dev: ETH0 I1121 21:51:56.446362 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:51:56.446378 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:51:57.445162 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:51:57.445453 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:51:57.445465 3798194 disk_worker.go:723] disk space is not compliant W1121 21:51:57.445468 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:51:57.447636 3798194 disk_worker.go:509] system disk:vda1 I1121 21:51:57.447676 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:51:58.445984 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:51:59.450030 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:51:59.450134 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:51:59.450160 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:51:59.471710 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:51:59.966526 3798194 disk_info.go:125] begin check local disk info of client I1121 21:51:59.970063 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:51:59.970076 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000347340 0xc000347380] E1121 21:52:06.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:52:06.411982 3798194 memory.go:184] no items to output this cycle I1121 21:52:06.412147 3798194 cpu.go:275] no items to output this cycle E1121 21:52:16.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:52:16.413357 3798194 memory.go:184] no items to output this cycle I1121 21:52:16.413611 3798194 cpu.go:275] no items to output this cycle E1121 21:52:26.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:52:26.411956 3798194 memory.go:191] Add success. I1121 21:52:26.412329 3798194 cpu.go:282] Add success. I1121 21:52:26.441943 3798194 net.go:642] Add success. I1121 21:52:26.446199 3798194 net.go:778] primary dev: ETH0 I1121 21:52:26.446215 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:52:26.446228 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:52:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:52:29.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:52:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:52:36.412093 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:52:36.412131 3798194 memory.go:184] no items to output this cycle I1121 21:52:36.412287 3798194 cpu.go:275] no items to output this cycle E1121 21:52:46.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:52:46.411988 3798194 memory.go:184] no items to output this cycle I1121 21:52:46.412152 3798194 cpu.go:275] no items to output this cycle E1121 21:52:56.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:52:56.413271 3798194 memory.go:191] Add success. W1121 21:52:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:52:56.413317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:52:56.413320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:52:56.417824 3798194 cpu.go:282] Add success. I1121 21:52:56.441323 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:52:56.441785 3798194 net.go:642] Add success. I1121 21:52:56.443320 3798194 net.go:778] primary dev: ETH0 I1121 21:52:56.443337 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:52:56.443349 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:52:57.452180 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:52:57.452344 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:52:57.452398 3798194 disk_worker.go:723] disk space is not compliant W1121 21:52:57.452401 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:52:57.460737 3798194 disk_worker.go:509] system disk:vda1 I1121 21:52:57.460779 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:52:58.446194 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:52:59.450187 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:52:59.450273 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:52:59.450302 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:52:59.470727 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:52:59.970176 3798194 disk_info.go:125] begin check local disk info of client I1121 21:52:59.974004 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:52:59.974013 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00039c080 0xc00039c0c0] E1121 21:53:06.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:53:06.412253 3798194 memory.go:184] no items to output this cycle I1121 21:53:06.412393 3798194 cpu.go:275] no items to output this cycle E1121 21:53:16.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:53:16.412259 3798194 memory.go:184] no items to output this cycle I1121 21:53:16.412406 3798194 cpu.go:275] no items to output this cycle E1121 21:53:26.411936 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:53:26.411976 3798194 memory.go:191] Add success. I1121 21:53:26.412355 3798194 cpu.go:282] Add success. I1121 21:53:26.442084 3798194 net.go:642] Add success. I1121 21:53:26.447922 3798194 net.go:778] primary dev: ETH0 I1121 21:53:26.447937 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:53:26.447952 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:53:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:53:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:53:29.450145 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:53:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:53:36.411976 3798194 memory.go:184] no items to output this cycle I1121 21:53:36.412131 3798194 cpu.go:275] no items to output this cycle E1121 21:53:46.413250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:53:46.413277 3798194 memory.go:184] no items to output this cycle I1121 21:53:46.413425 3798194 cpu.go:275] no items to output this cycle E1121 21:53:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:53:56.411956 3798194 memory.go:191] Add success. W1121 21:53:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:53:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:53:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:53:56.412353 3798194 cpu.go:282] Add success. I1121 21:53:56.440930 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:53:56.441534 3798194 net.go:642] Add success. I1121 21:53:56.445610 3798194 net.go:778] primary dev: ETH0 I1121 21:53:56.445625 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:53:56.445636 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:53:57.446377 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:53:57.446590 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:53:57.446662 3798194 disk_worker.go:723] disk space is not compliant W1121 21:53:57.446665 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:53:57.452620 3798194 disk_worker.go:509] system disk:vda1 I1121 21:53:57.452663 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:53:58.446371 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:53:59.450466 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:53:59.450549 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:53:59.450573 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:53:59.471087 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:53:59.975154 3798194 disk_info.go:125] begin check local disk info of client I1121 21:53:59.981050 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:53:59.981062 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000306cc0 0xc000306d00] E1121 21:54:06.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:54:06.413258 3798194 memory.go:184] no items to output this cycle I1121 21:54:06.413400 3798194 cpu.go:275] no items to output this cycle E1121 21:54:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:54:16.413271 3798194 memory.go:184] no items to output this cycle I1121 21:54:16.413422 3798194 cpu.go:275] no items to output this cycle I1121 21:54:17.563028 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:54:17.563041 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:54:26.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:54:26.412264 3798194 memory.go:191] Add success. I1121 21:54:26.416485 3798194 cpu.go:282] Add success. I1121 21:54:26.442323 3798194 net.go:642] Add success. I1121 21:54:26.443946 3798194 net.go:778] primary dev: ETH0 I1121 21:54:26.443961 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:54:26.443981 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:54:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:54:29.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:54:29.450511 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:54:36.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:54:36.413255 3798194 memory.go:184] no items to output this cycle I1121 21:54:36.413398 3798194 cpu.go:275] no items to output this cycle E1121 21:54:46.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:54:46.412257 3798194 memory.go:184] no items to output this cycle I1121 21:54:46.412407 3798194 cpu.go:275] no items to output this cycle E1121 21:54:56.412869 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:54:56.412911 3798194 memory.go:191] Add success. W1121 21:54:56.412939 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:54:56.412952 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:54:56.412955 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:54:56.418100 3798194 cpu.go:282] Add success. I1121 21:54:56.441419 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:54:56.441905 3798194 net.go:642] Add success. I1121 21:54:56.443423 3798194 net.go:778] primary dev: ETH0 I1121 21:54:56.443438 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:54:56.443451 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:54:57.445151 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:54:57.445346 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:54:57.445407 3798194 disk_worker.go:723] disk space is not compliant W1121 21:54:57.445411 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:54:57.447502 3798194 disk_worker.go:509] system disk:vda1 I1121 21:54:57.447546 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:54:58.445983 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:54:59.450489 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:54:59.450569 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:54:59.450594 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:54:59.471089 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:54:59.981166 3798194 disk_info.go:125] begin check local disk info of client I1121 21:54:59.985065 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:54:59.985075 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a1100 0xc0003a1140] E1121 21:55:06.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:55:06.413254 3798194 memory.go:184] no items to output this cycle I1121 21:55:06.413415 3798194 cpu.go:275] no items to output this cycle E1121 21:55:16.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:55:16.413255 3798194 memory.go:184] no items to output this cycle I1121 21:55:16.413401 3798194 cpu.go:275] no items to output this cycle E1121 21:55:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:55:26.413268 3798194 memory.go:191] Add success. I1121 21:55:26.417540 3798194 cpu.go:282] Add success. I1121 21:55:26.442072 3798194 net.go:642] Add success. I1121 21:55:26.443793 3798194 net.go:778] primary dev: ETH0 I1121 21:55:26.443807 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:55:26.443819 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:55:29.450543 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:55:29.450623 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:55:29.450651 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:55:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:55:36.412258 3798194 memory.go:184] no items to output this cycle I1121 21:55:36.412401 3798194 cpu.go:275] no items to output this cycle E1121 21:55:46.412613 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:55:46.412636 3798194 memory.go:184] no items to output this cycle I1121 21:55:46.412778 3798194 cpu.go:275] no items to output this cycle E1121 21:55:56.412493 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:55:56.412527 3798194 memory.go:191] Add success. W1121 21:55:56.412553 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:55:56.412565 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:55:56.412568 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:55:56.416608 3798194 cpu.go:282] Add success. I1121 21:55:56.443640 3798194 event_worker.go:152] Polling the log file for events... I1121 21:55:56.443853 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:55:56.444366 3798194 net.go:642] Add success. I1121 21:55:56.445769 3798194 net.go:778] primary dev: ETH0 I1121 21:55:56.445783 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:55:56.445794 3798194 net.go:659] Add eni metrics success, count: 7 E1121 21:55:57.450794 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 21:55:57.450808 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:55:57.451015 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:55:57.451025 3798194 disk_worker.go:723] disk space is not compliant W1121 21:55:57.451028 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:55:57.461396 3798194 disk_worker.go:509] system disk:vda1 I1121 21:55:57.461442 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 21:55:58.448148 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 21:55:58.448163 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 21:55:59.448970 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 21:55:59.450043 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:55:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:55:59.450123 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:55:59.471630 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:55:59.985176 3798194 disk_info.go:125] begin check local disk info of client I1121 21:55:59.988700 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:55:59.988708 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a1400 0xc0003a1440] E1121 21:56:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:56:06.411985 3798194 memory.go:184] no items to output this cycle I1121 21:56:06.412145 3798194 cpu.go:275] no items to output this cycle E1121 21:56:16.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:56:16.412266 3798194 memory.go:184] no items to output this cycle I1121 21:56:16.412408 3798194 cpu.go:275] no items to output this cycle E1121 21:56:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:56:26.411953 3798194 memory.go:191] Add success. I1121 21:56:26.412307 3798194 cpu.go:282] Add success. I1121 21:56:26.441930 3798194 net.go:642] Add success. I1121 21:56:26.445654 3798194 net.go:778] primary dev: ETH0 I1121 21:56:26.445669 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:56:26.445681 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:56:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:56:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:56:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:56:36.412175 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:56:36.412198 3798194 memory.go:184] no items to output this cycle I1121 21:56:36.412342 3798194 cpu.go:275] no items to output this cycle E1121 21:56:46.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:56:46.411994 3798194 memory.go:184] no items to output this cycle I1121 21:56:46.412159 3798194 cpu.go:275] no items to output this cycle E1121 21:56:56.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:56:56.412273 3798194 memory.go:191] Add success. W1121 21:56:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:56:56.412317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:56:56.412320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:56:56.416536 3798194 cpu.go:282] Add success. I1121 21:56:56.441940 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:56:56.442541 3798194 net.go:642] Add success. I1121 21:56:56.443913 3798194 net.go:778] primary dev: ETH0 I1121 21:56:56.443927 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:56:56.443941 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:56:57.451683 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:56:57.451896 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:56:57.451907 3798194 disk_worker.go:723] disk space is not compliant W1121 21:56:57.451910 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:56:57.462616 3798194 disk_worker.go:509] system disk:vda1 I1121 21:56:57.462659 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:56:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:56:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:56:59.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:56:59.450505 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:56:59.471012 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:56:59.988809 3798194 disk_info.go:125] begin check local disk info of client I1121 21:56:59.992688 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:56:59.992698 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a0680 0xc0003a06c0] E1121 21:57:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:57:06.411982 3798194 memory.go:184] no items to output this cycle I1121 21:57:06.412147 3798194 cpu.go:275] no items to output this cycle E1121 21:57:16.412338 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:57:16.412362 3798194 memory.go:184] no items to output this cycle I1121 21:57:16.412509 3798194 cpu.go:275] no items to output this cycle I1121 21:57:17.564029 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 21:57:17.564038 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 21:57:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:57:26.413268 3798194 memory.go:191] Add success. I1121 21:57:26.417494 3798194 cpu.go:282] Add success. I1121 21:57:26.442335 3798194 net.go:642] Add success. I1121 21:57:26.443793 3798194 net.go:778] primary dev: ETH0 I1121 21:57:26.443807 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:57:26.443822 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:57:29.449676 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:57:29.449754 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:57:29.449787 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:57:36.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:57:36.412885 3798194 memory.go:184] no items to output this cycle I1121 21:57:36.413031 3798194 cpu.go:275] no items to output this cycle E1121 21:57:46.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:57:46.411985 3798194 memory.go:184] no items to output this cycle I1121 21:57:46.412149 3798194 cpu.go:275] no items to output this cycle E1121 21:57:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:57:56.412264 3798194 memory.go:191] Add success. W1121 21:57:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:57:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:57:56.412314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:57:56.416494 3798194 cpu.go:282] Add success. I1121 21:57:56.440970 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:57:56.441587 3798194 net.go:642] Add success. I1121 21:57:56.443241 3798194 net.go:778] primary dev: ETH0 I1121 21:57:56.443255 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:57:56.443272 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:57:57.445600 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:57:57.445877 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:57:57.445890 3798194 disk_worker.go:723] disk space is not compliant W1121 21:57:57.445893 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:57:57.450323 3798194 disk_worker.go:509] system disk:vda1 I1121 21:57:57.450362 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:57:58.446383 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:57:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:57:59.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:57:59.450507 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:57:59.471027 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:57:59.994391 3798194 disk_info.go:125] begin check local disk info of client I1121 21:57:59.998325 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:57:59.998334 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004bee00 0xc0004bee40] E1121 21:58:06.411973 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:58:06.411996 3798194 memory.go:184] no items to output this cycle I1121 21:58:06.412149 3798194 cpu.go:275] no items to output this cycle E1121 21:58:16.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:58:16.411993 3798194 memory.go:184] no items to output this cycle I1121 21:58:16.412149 3798194 cpu.go:275] no items to output this cycle E1121 21:58:26.412308 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:58:26.412341 3798194 memory.go:191] Add success. I1121 21:58:26.416814 3798194 cpu.go:282] Add success. I1121 21:58:26.441284 3798194 net.go:642] Add success. I1121 21:58:26.442786 3798194 net.go:778] primary dev: ETH0 I1121 21:58:26.442799 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:58:26.442813 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:58:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:58:29.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:58:29.450148 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:58:36.412005 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:58:36.412031 3798194 memory.go:184] no items to output this cycle I1121 21:58:36.412189 3798194 cpu.go:275] no items to output this cycle E1121 21:58:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:58:46.411983 3798194 memory.go:184] no items to output this cycle I1121 21:58:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 21:58:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:58:56.411946 3798194 memory.go:191] Add success. W1121 21:58:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:58:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:58:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:58:56.412344 3798194 cpu.go:282] Add success. I1121 21:58:56.441816 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:58:56.442401 3798194 net.go:642] Add success. I1121 21:58:56.446329 3798194 net.go:778] primary dev: ETH0 I1121 21:58:56.446343 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:58:56.446357 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:58:57.450366 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:58:57.450540 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:58:57.450600 3798194 disk_worker.go:723] disk space is not compliant W1121 21:58:57.450603 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:58:57.458712 3798194 disk_worker.go:509] system disk:vda1 I1121 21:58:57.458755 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:58:58.446010 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:58:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:58:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:58:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:58:59.471686 3798194 disk_local_worker.go:436] Get disk info: [] I1121 21:58:59.998428 3798194 disk_info.go:125] begin check local disk info of client I1121 21:59:00.001950 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 21:59:00.001958 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a5a80 0xc0003a5ac0] E1121 21:59:06.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:59:06.412262 3798194 memory.go:184] no items to output this cycle I1121 21:59:06.412407 3798194 cpu.go:275] no items to output this cycle E1121 21:59:16.411939 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:59:16.411962 3798194 memory.go:184] no items to output this cycle I1121 21:59:16.412128 3798194 cpu.go:275] no items to output this cycle E1121 21:59:26.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:59:26.411947 3798194 memory.go:191] Add success. I1121 21:59:26.412309 3798194 cpu.go:282] Add success. I1121 21:59:26.442099 3798194 net.go:642] Add success. I1121 21:59:26.446338 3798194 net.go:778] primary dev: ETH0 I1121 21:59:26.446355 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:59:26.446371 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:59:29.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:59:29.450122 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:59:29.450159 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 21:59:36.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:59:36.413269 3798194 memory.go:184] no items to output this cycle I1121 21:59:36.413417 3798194 cpu.go:275] no items to output this cycle E1121 21:59:46.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:59:46.412265 3798194 memory.go:184] no items to output this cycle I1121 21:59:46.412426 3798194 cpu.go:275] no items to output this cycle E1121 21:59:56.411932 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 21:59:56.411966 3798194 memory.go:191] Add success. W1121 21:59:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 21:59:56.412006 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 21:59:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 21:59:56.412385 3798194 cpu.go:282] Add success. I1121 21:59:56.441949 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 21:59:56.442600 3798194 net.go:642] Add success. I1121 21:59:56.446732 3798194 net.go:778] primary dev: ETH0 I1121 21:59:56.446747 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 21:59:56.446761 3798194 net.go:659] Add eni metrics success, count: 7 I1121 21:59:57.453424 3798194 custom_config.go:106] query custom config with name: gpu W1121 21:59:57.453632 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 21:59:57.453643 3798194 disk_worker.go:723] disk space is not compliant W1121 21:59:57.453646 3798194 disk_worker.go:743] disk inode is not compliant I1121 21:59:57.461868 3798194 disk_worker.go:509] system disk:vda1 I1121 21:59:57.461916 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 21:59:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 21:59:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 21:59:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 21:59:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 21:59:59.471653 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:00:00.002065 3798194 disk_info.go:125] begin check local disk info of client I1121 22:00:00.005631 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:00:00.005640 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040dec0 0xc00040df00] E1121 22:00:06.413191 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:00:06.413218 3798194 memory.go:184] no items to output this cycle I1121 22:00:06.413368 3798194 cpu.go:275] no items to output this cycle E1121 22:00:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:00:16.411980 3798194 memory.go:184] no items to output this cycle I1121 22:00:16.412136 3798194 cpu.go:275] no items to output this cycle I1121 22:00:17.564200 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:00:17.564209 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:00:26.411320 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:00:26.411353 3798194 memory.go:191] Add success. I1121 22:00:26.411696 3798194 cpu.go:282] Add success. I1121 22:00:26.441300 3798194 net.go:642] Add success. I1121 22:00:26.445358 3798194 net.go:778] primary dev: ETH0 I1121 22:00:26.445374 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:00:26.445387 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:00:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:00:29.450127 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:00:29.450153 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:00:36.412805 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:00:36.412827 3798194 memory.go:184] no items to output this cycle I1121 22:00:36.412969 3798194 cpu.go:275] no items to output this cycle E1121 22:00:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:00:46.411979 3798194 memory.go:184] no items to output this cycle I1121 22:00:46.412143 3798194 cpu.go:275] no items to output this cycle E1121 22:00:56.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:00:56.412889 3798194 memory.go:191] Add success. W1121 22:00:56.412915 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:00:56.412926 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:00:56.412929 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:00:56.417244 3798194 cpu.go:282] Add success. I1121 22:00:56.445734 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:00:56.446707 3798194 net.go:642] Add success. I1121 22:00:56.448483 3798194 net.go:778] primary dev: ETH0 I1121 22:00:56.448499 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:00:56.448516 3798194 net.go:659] Add eni metrics success, count: 7 E1121 22:00:57.450733 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 22:00:57.450747 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:00:57.450941 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:00:57.450951 3798194 disk_worker.go:723] disk space is not compliant W1121 22:00:57.450955 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:00:57.459228 3798194 disk_worker.go:509] system disk:vda1 I1121 22:00:57.459270 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 22:00:58.447893 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 22:00:58.447908 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 22:00:59.448602 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 22:00:59.449676 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:00:59.449726 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:00:59.449744 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:00:59.471263 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:01:00.005746 3798194 disk_info.go:125] begin check local disk info of client I1121 22:01:00.009405 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:01:00.009414 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004e4400 0xc0004e4440] E1121 22:01:06.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:01:06.411984 3798194 memory.go:184] no items to output this cycle I1121 22:01:06.412142 3798194 cpu.go:275] no items to output this cycle E1121 22:01:16.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:01:16.413257 3798194 memory.go:184] no items to output this cycle I1121 22:01:16.413401 3798194 cpu.go:275] no items to output this cycle E1121 22:01:26.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:01:26.412276 3798194 memory.go:191] Add success. I1121 22:01:26.424045 3798194 cpu.go:282] Add success. I1121 22:01:26.441705 3798194 net.go:642] Add success. I1121 22:01:26.443218 3798194 net.go:778] primary dev: ETH0 I1121 22:01:26.443234 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:01:26.443247 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:01:29.449907 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:01:29.449985 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:01:29.450012 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:01:36.411945 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:01:36.411967 3798194 memory.go:184] no items to output this cycle I1121 22:01:36.412124 3798194 cpu.go:275] no items to output this cycle E1121 22:01:46.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:01:46.413272 3798194 memory.go:184] no items to output this cycle I1121 22:01:46.413425 3798194 cpu.go:275] no items to output this cycle E1121 22:01:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:01:56.411954 3798194 memory.go:191] Add success. W1121 22:01:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:01:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:01:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:01:56.412356 3798194 cpu.go:282] Add success. I1121 22:01:56.441846 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:01:56.442444 3798194 net.go:642] Add success. I1121 22:01:56.450526 3798194 net.go:778] primary dev: ETH0 I1121 22:01:56.450544 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:01:56.450556 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:01:57.445306 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:01:57.445521 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:01:57.445587 3798194 disk_worker.go:723] disk space is not compliant W1121 22:01:57.445590 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:01:57.450577 3798194 disk_worker.go:509] system disk:vda1 I1121 22:01:57.450617 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:01:58.445987 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:01:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:01:59.450124 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:01:59.450154 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:01:59.471645 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:02:00.009534 3798194 disk_info.go:125] begin check local disk info of client I1121 22:02:00.013083 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:02:00.013092 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000456200 0xc000456240] E1121 22:02:06.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:02:06.412262 3798194 memory.go:184] no items to output this cycle I1121 22:02:06.412405 3798194 cpu.go:275] no items to output this cycle E1121 22:02:16.412537 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:02:16.412559 3798194 memory.go:184] no items to output this cycle I1121 22:02:16.412706 3798194 cpu.go:275] no items to output this cycle E1121 22:02:26.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:02:26.411964 3798194 memory.go:191] Add success. I1121 22:02:26.412318 3798194 cpu.go:282] Add success. I1121 22:02:26.441959 3798194 net.go:642] Add success. I1121 22:02:26.445975 3798194 net.go:778] primary dev: ETH0 I1121 22:02:26.445990 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:02:26.446004 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:02:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:02:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:02:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:02:36.413256 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:02:36.413278 3798194 memory.go:184] no items to output this cycle I1121 22:02:36.413420 3798194 cpu.go:275] no items to output this cycle E1121 22:02:46.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:02:46.411986 3798194 memory.go:184] no items to output this cycle I1121 22:02:46.412146 3798194 cpu.go:275] no items to output this cycle E1121 22:02:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:02:56.411951 3798194 memory.go:191] Add success. W1121 22:02:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:02:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:02:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:02:56.412357 3798194 cpu.go:282] Add success. I1121 22:02:56.441812 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:02:56.442318 3798194 net.go:642] Add success. I1121 22:02:56.446512 3798194 net.go:778] primary dev: ETH0 I1121 22:02:56.446527 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:02:56.446540 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:02:57.450746 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:02:57.450905 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:02:57.450956 3798194 disk_worker.go:723] disk space is not compliant W1121 22:02:57.450959 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:02:57.459350 3798194 disk_worker.go:509] system disk:vda1 I1121 22:02:57.459393 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:02:58.446013 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:02:59.450030 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:02:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:02:59.450144 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:02:59.471705 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:03:00.013185 3798194 disk_info.go:125] begin check local disk info of client I1121 22:03:00.016797 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:03:00.016806 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00034d700 0xc00034d740] E1121 22:03:06.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:03:06.412878 3798194 memory.go:184] no items to output this cycle I1121 22:03:06.413025 3798194 cpu.go:275] no items to output this cycle E1121 22:03:16.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:03:16.413256 3798194 memory.go:184] no items to output this cycle I1121 22:03:16.413398 3798194 cpu.go:275] no items to output this cycle I1121 22:03:17.564372 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:03:17.564391 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:03:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:03:26.413264 3798194 memory.go:191] Add success. I1121 22:03:26.417346 3798194 cpu.go:282] Add success. I1121 22:03:26.442080 3798194 net.go:642] Add success. I1121 22:03:26.443576 3798194 net.go:778] primary dev: ETH0 I1121 22:03:26.443593 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:03:26.443706 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:03:29.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:03:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:03:29.450511 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:03:36.411993 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:03:36.412018 3798194 memory.go:184] no items to output this cycle I1121 22:03:36.412180 3798194 cpu.go:275] no items to output this cycle E1121 22:03:46.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:03:46.411992 3798194 memory.go:184] no items to output this cycle I1121 22:03:46.412166 3798194 cpu.go:275] no items to output this cycle E1121 22:03:56.411017 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:03:56.411052 3798194 memory.go:191] Add success. W1121 22:03:56.411078 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:03:56.411091 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:03:56.411095 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:03:56.411446 3798194 cpu.go:282] Add success. I1121 22:03:56.440961 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:03:56.441542 3798194 net.go:642] Add success. I1121 22:03:56.445399 3798194 net.go:778] primary dev: ETH0 I1121 22:03:56.445414 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:03:56.445426 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:03:57.450702 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:03:57.450862 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:03:57.450916 3798194 disk_worker.go:723] disk space is not compliant W1121 22:03:57.450920 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:03:57.459210 3798194 disk_worker.go:509] system disk:vda1 I1121 22:03:57.459254 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:03:58.446385 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:03:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:03:59.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:03:59.450504 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:03:59.471017 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:04:00.024169 3798194 disk_info.go:125] begin check local disk info of client I1121 22:04:00.029803 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:04:00.029815 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000381b40 0xc000381b80] E1121 22:04:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:04:06.411983 3798194 memory.go:184] no items to output this cycle I1121 22:04:06.412146 3798194 cpu.go:275] no items to output this cycle E1121 22:04:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:04:16.411978 3798194 memory.go:184] no items to output this cycle I1121 22:04:16.412136 3798194 cpu.go:275] no items to output this cycle E1121 22:04:26.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:04:26.412276 3798194 memory.go:191] Add success. I1121 22:04:26.416299 3798194 cpu.go:282] Add success. I1121 22:04:26.442123 3798194 net.go:642] Add success. I1121 22:04:26.443597 3798194 net.go:778] primary dev: ETH0 I1121 22:04:26.443610 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:04:26.443624 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:04:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:04:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:04:29.450124 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:04:36.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:04:36.411980 3798194 memory.go:184] no items to output this cycle I1121 22:04:36.412138 3798194 cpu.go:275] no items to output this cycle E1121 22:04:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:04:46.411985 3798194 memory.go:184] no items to output this cycle I1121 22:04:46.412144 3798194 cpu.go:275] no items to output this cycle E1121 22:04:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:04:56.411950 3798194 memory.go:191] Add success. W1121 22:04:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:04:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:04:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:04:56.412351 3798194 cpu.go:282] Add success. I1121 22:04:56.440958 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:04:56.441547 3798194 net.go:642] Add success. I1121 22:04:56.446051 3798194 net.go:778] primary dev: ETH0 I1121 22:04:56.446068 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:04:56.446080 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:04:57.450697 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:04:57.450865 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:04:57.450918 3798194 disk_worker.go:723] disk space is not compliant W1121 22:04:57.450922 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:04:57.459250 3798194 disk_worker.go:509] system disk:vda1 I1121 22:04:57.459292 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:04:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:04:59.449999 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:04:59.450081 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:04:59.450105 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:04:59.471648 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:05:00.032143 3798194 disk_info.go:125] begin check local disk info of client I1121 22:05:00.035676 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:05:00.035685 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e580 0xc00030e5c0] E1121 22:05:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:05:06.412252 3798194 memory.go:184] no items to output this cycle I1121 22:05:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 22:05:16.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:05:16.412251 3798194 memory.go:184] no items to output this cycle I1121 22:05:16.412398 3798194 cpu.go:275] no items to output this cycle E1121 22:05:26.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:05:26.412280 3798194 memory.go:191] Add success. I1121 22:05:26.416554 3798194 cpu.go:282] Add success. I1121 22:05:26.442078 3798194 net.go:642] Add success. I1121 22:05:26.443586 3798194 net.go:778] primary dev: ETH0 I1121 22:05:26.443602 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:05:26.443614 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:05:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:05:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:05:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:05:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:05:36.412259 3798194 memory.go:184] no items to output this cycle I1121 22:05:36.412402 3798194 cpu.go:275] no items to output this cycle E1121 22:05:46.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:05:46.413270 3798194 memory.go:184] no items to output this cycle I1121 22:05:46.413423 3798194 cpu.go:275] no items to output this cycle E1121 22:05:56.413072 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:05:56.413106 3798194 memory.go:191] Add success. W1121 22:05:56.413155 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:05:56.413167 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:05:56.413170 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:05:56.417423 3798194 cpu.go:282] Add success. I1121 22:05:56.441575 3798194 event_worker.go:152] Polling the log file for events... I1121 22:05:56.444867 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:05:56.445346 3798194 net.go:642] Add success. I1121 22:05:56.446949 3798194 net.go:778] primary dev: ETH0 I1121 22:05:56.446963 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:05:56.446975 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:05:56.446992 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795268156, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 22:05:56.457911 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 22:05:57.446452 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 22:05:57.446473 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:05:57.446669 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:05:57.446740 3798194 disk_worker.go:723] disk space is not compliant W1121 22:05:57.446743 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:05:57.449554 3798194 disk_worker.go:509] system disk:vda1 I1121 22:05:57.449598 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 22:05:58.448327 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 22:05:58.448351 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 22:05:59.448626 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 22:05:59.449658 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:05:59.449710 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:05:59.449735 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:05:59.471205 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:06:00.035789 3798194 disk_info.go:125] begin check local disk info of client I1121 22:06:00.039746 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:06:00.039755 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000416600 0xc000416640] E1121 22:06:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:06:06.411979 3798194 memory.go:184] no items to output this cycle I1121 22:06:06.412133 3798194 cpu.go:275] no items to output this cycle E1121 22:06:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:06:16.411984 3798194 memory.go:184] no items to output this cycle I1121 22:06:16.412135 3798194 cpu.go:275] no items to output this cycle I1121 22:06:17.564553 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:06:17.564562 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:06:26.412869 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:06:26.412904 3798194 memory.go:191] Add success. I1121 22:06:26.417135 3798194 cpu.go:282] Add success. I1121 22:06:26.441653 3798194 net.go:642] Add success. I1121 22:06:26.443202 3798194 net.go:778] primary dev: ETH0 I1121 22:06:26.443216 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:06:26.443230 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:06:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:06:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:06:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:06:36.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:06:36.411978 3798194 memory.go:184] no items to output this cycle I1121 22:06:36.412138 3798194 cpu.go:275] no items to output this cycle E1121 22:06:46.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:06:46.411985 3798194 memory.go:184] no items to output this cycle I1121 22:06:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 22:06:56.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:06:56.412881 3798194 memory.go:191] Add success. W1121 22:06:56.412926 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:06:56.412937 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:06:56.412940 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:06:56.417402 3798194 cpu.go:282] Add success. I1121 22:06:56.441789 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:06:56.442300 3798194 net.go:642] Add success. I1121 22:06:56.443951 3798194 net.go:778] primary dev: ETH0 I1121 22:06:56.443965 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:06:56.443978 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:06:57.451466 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:06:57.451674 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:06:57.451685 3798194 disk_worker.go:723] disk space is not compliant W1121 22:06:57.451688 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:06:57.460255 3798194 disk_worker.go:509] system disk:vda1 I1121 22:06:57.460296 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:06:58.445692 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:06:59.450647 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:06:59.450736 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:06:59.450761 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:06:59.471266 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:07:00.039848 3798194 disk_info.go:125] begin check local disk info of client I1121 22:07:00.043726 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:07:00.043735 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00039b9c0 0xc00039ba00] E1121 22:07:06.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:07:06.413253 3798194 memory.go:184] no items to output this cycle I1121 22:07:06.413403 3798194 cpu.go:275] no items to output this cycle E1121 22:07:16.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:07:16.411991 3798194 memory.go:184] no items to output this cycle I1121 22:07:16.412154 3798194 cpu.go:275] no items to output this cycle E1121 22:07:26.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:07:26.413281 3798194 memory.go:191] Add success. I1121 22:07:26.417870 3798194 cpu.go:282] Add success. I1121 22:07:26.441781 3798194 net.go:642] Add success. I1121 22:07:26.443452 3798194 net.go:778] primary dev: ETH0 I1121 22:07:26.443469 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:07:26.443489 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:07:29.449993 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:07:29.450066 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:07:29.450094 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:07:36.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:07:36.412877 3798194 memory.go:184] no items to output this cycle I1121 22:07:36.413015 3798194 cpu.go:275] no items to output this cycle E1121 22:07:46.413510 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:07:46.413531 3798194 memory.go:184] no items to output this cycle I1121 22:07:46.413680 3798194 cpu.go:275] no items to output this cycle E1121 22:07:56.412829 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:07:56.412862 3798194 memory.go:191] Add success. W1121 22:07:56.412895 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:07:56.412906 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:07:56.412910 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:07:56.417317 3798194 cpu.go:282] Add success. I1121 22:07:56.441732 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:07:56.442227 3798194 net.go:642] Add success. I1121 22:07:56.443902 3798194 net.go:778] primary dev: ETH0 I1121 22:07:56.443917 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:07:56.443929 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:07:57.454844 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:07:57.455006 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:07:57.455059 3798194 disk_worker.go:723] disk space is not compliant W1121 22:07:57.455063 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:07:57.467457 3798194 disk_worker.go:509] system disk:vda1 I1121 22:07:57.467499 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:07:58.446530 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:07:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:07:59.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:07:59.450160 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:07:59.471694 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:08:00.043835 3798194 disk_info.go:125] begin check local disk info of client I1121 22:08:00.047735 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:08:00.047745 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00039bbc0 0xc00039bc00] E1121 22:08:06.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:08:06.413251 3798194 memory.go:184] no items to output this cycle I1121 22:08:06.413393 3798194 cpu.go:275] no items to output this cycle E1121 22:08:16.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:08:16.411991 3798194 memory.go:184] no items to output this cycle I1121 22:08:16.412163 3798194 cpu.go:275] no items to output this cycle E1121 22:08:26.411934 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:08:26.411966 3798194 memory.go:191] Add success. I1121 22:08:26.412310 3798194 cpu.go:282] Add success. I1121 22:08:26.441940 3798194 net.go:642] Add success. I1121 22:08:26.445881 3798194 net.go:778] primary dev: ETH0 I1121 22:08:26.445897 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:08:26.445909 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:08:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:08:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:08:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:08:36.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:08:36.412260 3798194 memory.go:184] no items to output this cycle I1121 22:08:36.412410 3798194 cpu.go:275] no items to output this cycle E1121 22:08:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:08:46.412259 3798194 memory.go:184] no items to output this cycle I1121 22:08:46.412414 3798194 cpu.go:275] no items to output this cycle E1121 22:08:56.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:08:56.413016 3798194 memory.go:191] Add success. W1121 22:08:56.413054 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:08:56.413067 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:08:56.413070 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:08:56.417810 3798194 cpu.go:282] Add success. I1121 22:08:56.441748 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:08:56.442369 3798194 net.go:642] Add success. I1121 22:08:56.443894 3798194 net.go:778] primary dev: ETH0 I1121 22:08:56.443910 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:08:56.443924 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:08:57.445915 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:08:57.446202 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:08:57.446215 3798194 disk_worker.go:723] disk space is not compliant W1121 22:08:57.446218 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:08:57.451250 3798194 disk_worker.go:509] system disk:vda1 I1121 22:08:57.451290 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:08:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:08:59.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:08:59.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:08:59.450505 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:08:59.474622 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:09:00.050144 3798194 disk_info.go:125] begin check local disk info of client I1121 22:09:00.053994 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:09:00.054004 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ef80 0xc00030efc0] E1121 22:09:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:09:06.412257 3798194 memory.go:184] no items to output this cycle I1121 22:09:06.412401 3798194 cpu.go:275] no items to output this cycle E1121 22:09:16.412339 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:09:16.412362 3798194 memory.go:184] no items to output this cycle I1121 22:09:16.412508 3798194 cpu.go:275] no items to output this cycle I1121 22:09:17.565649 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:09:17.565657 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:09:26.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:09:26.412892 3798194 memory.go:191] Add success. I1121 22:09:26.417420 3798194 cpu.go:282] Add success. I1121 22:09:26.442079 3798194 net.go:642] Add success. I1121 22:09:26.443624 3798194 net.go:778] primary dev: ETH0 I1121 22:09:26.443637 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:09:26.443651 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:09:29.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:09:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:09:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:09:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:09:36.412256 3798194 memory.go:184] no items to output this cycle I1121 22:09:36.412404 3798194 cpu.go:275] no items to output this cycle E1121 22:09:46.412246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:09:46.412266 3798194 memory.go:184] no items to output this cycle I1121 22:09:46.412413 3798194 cpu.go:275] no items to output this cycle E1121 22:09:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:09:56.411955 3798194 memory.go:191] Add success. W1121 22:09:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:09:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:09:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:09:56.412362 3798194 cpu.go:282] Add success. I1121 22:09:56.440924 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:09:56.441543 3798194 net.go:642] Add success. I1121 22:09:56.446261 3798194 net.go:778] primary dev: ETH0 I1121 22:09:56.446277 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:09:56.446291 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:09:57.445499 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:09:57.445757 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:09:57.445769 3798194 disk_worker.go:723] disk space is not compliant W1121 22:09:57.445772 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:09:57.447882 3798194 disk_worker.go:509] system disk:vda1 I1121 22:09:57.447920 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:09:58.446369 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:09:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:09:59.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:09:59.450512 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:09:59.471058 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:10:00.054107 3798194 disk_info.go:125] begin check local disk info of client I1121 22:10:00.057970 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:10:00.057991 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00049e1c0 0xc00049e200] E1121 22:10:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:10:06.411978 3798194 memory.go:184] no items to output this cycle I1121 22:10:06.412129 3798194 cpu.go:275] no items to output this cycle E1121 22:10:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:10:16.412259 3798194 memory.go:184] no items to output this cycle I1121 22:10:16.412408 3798194 cpu.go:275] no items to output this cycle E1121 22:10:26.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:10:26.412267 3798194 memory.go:191] Add success. I1121 22:10:26.416371 3798194 cpu.go:282] Add success. I1121 22:10:26.442044 3798194 net.go:642] Add success. I1121 22:10:26.443699 3798194 net.go:778] primary dev: ETH0 I1121 22:10:26.443715 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:10:26.443728 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:10:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:10:29.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:10:29.450126 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:10:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:10:36.412256 3798194 memory.go:184] no items to output this cycle I1121 22:10:36.412398 3798194 cpu.go:275] no items to output this cycle E1121 22:10:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:10:46.411979 3798194 memory.go:184] no items to output this cycle I1121 22:10:46.412139 3798194 cpu.go:275] no items to output this cycle E1121 22:10:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:10:56.411957 3798194 memory.go:191] Add success. W1121 22:10:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:10:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:10:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:10:56.412349 3798194 cpu.go:282] Add success. I1121 22:10:56.444556 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:10:56.445080 3798194 net.go:642] Add success. I1121 22:10:56.449898 3798194 net.go:778] primary dev: ETH0 I1121 22:10:56.449916 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:10:56.449928 3798194 net.go:659] Add eni metrics success, count: 7 E1121 22:10:57.450733 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 22:10:57.450747 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:10:57.450939 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:10:57.450948 3798194 disk_worker.go:723] disk space is not compliant W1121 22:10:57.450951 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:10:57.459085 3798194 disk_worker.go:509] system disk:vda1 I1121 22:10:57.459164 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 22:10:58.448487 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 22:10:58.448513 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 22:10:59.448985 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 22:10:59.450061 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:10:59.450119 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:10:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:10:59.471664 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:11:00.058082 3798194 disk_info.go:125] begin check local disk info of client I1121 22:11:00.061913 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:11:00.061922 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037f740 0xc00037f780] E1121 22:11:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:11:06.411985 3798194 memory.go:184] no items to output this cycle I1121 22:11:06.412149 3798194 cpu.go:275] no items to output this cycle E1121 22:11:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:11:16.411979 3798194 memory.go:184] no items to output this cycle I1121 22:11:16.412136 3798194 cpu.go:275] no items to output this cycle E1121 22:11:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:11:26.411952 3798194 memory.go:191] Add success. I1121 22:11:26.412302 3798194 cpu.go:282] Add success. I1121 22:11:26.441748 3798194 net.go:642] Add success. I1121 22:11:26.445854 3798194 net.go:778] primary dev: ETH0 I1121 22:11:26.445869 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:11:26.445883 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:11:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:11:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:11:29.450145 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:11:36.412904 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:11:36.412926 3798194 memory.go:184] no items to output this cycle I1121 22:11:36.413067 3798194 cpu.go:275] no items to output this cycle E1121 22:11:46.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:11:46.413269 3798194 memory.go:184] no items to output this cycle I1121 22:11:46.413413 3798194 cpu.go:275] no items to output this cycle E1121 22:11:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:11:56.413263 3798194 memory.go:191] Add success. W1121 22:11:56.413291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:11:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:11:56.413306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:11:56.417271 3798194 cpu.go:282] Add success. I1121 22:11:56.440947 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:11:56.441524 3798194 net.go:642] Add success. I1121 22:11:56.442981 3798194 net.go:778] primary dev: ETH0 I1121 22:11:56.442995 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:11:56.443007 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:11:57.450935 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:11:57.451103 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:11:57.451177 3798194 disk_worker.go:723] disk space is not compliant W1121 22:11:57.451180 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:11:57.459900 3798194 disk_worker.go:509] system disk:vda1 I1121 22:11:57.459943 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:11:58.446370 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:11:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:11:59.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:11:59.450505 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:11:59.471014 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:12:00.062026 3798194 disk_info.go:125] begin check local disk info of client I1121 22:12:00.065915 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:12:00.065925 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f31c0 0xc0003f3200] E1121 22:12:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:12:06.411985 3798194 memory.go:184] no items to output this cycle I1121 22:12:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 22:12:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:12:16.411978 3798194 memory.go:184] no items to output this cycle I1121 22:12:16.412142 3798194 cpu.go:275] no items to output this cycle I1121 22:12:17.566647 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:12:17.566656 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:12:26.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:12:26.413279 3798194 memory.go:191] Add success. I1121 22:12:26.417392 3798194 cpu.go:282] Add success. I1121 22:12:26.442037 3798194 net.go:642] Add success. I1121 22:12:26.443704 3798194 net.go:778] primary dev: ETH0 I1121 22:12:26.443719 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:12:26.443733 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:12:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:12:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:12:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:12:36.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:12:36.412877 3798194 memory.go:184] no items to output this cycle I1121 22:12:36.413021 3798194 cpu.go:275] no items to output this cycle E1121 22:12:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:12:46.411975 3798194 memory.go:184] no items to output this cycle I1121 22:12:46.412139 3798194 cpu.go:275] no items to output this cycle E1121 22:12:56.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:12:56.412265 3798194 memory.go:191] Add success. W1121 22:12:56.412298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:12:56.412310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:12:56.412314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:12:56.416636 3798194 cpu.go:282] Add success. I1121 22:12:56.441132 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:12:56.441704 3798194 net.go:642] Add success. I1121 22:12:56.443676 3798194 net.go:778] primary dev: ETH0 I1121 22:12:56.443693 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:12:56.443713 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:12:57.451050 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:12:57.451277 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:12:57.451287 3798194 disk_worker.go:723] disk space is not compliant W1121 22:12:57.451290 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:12:57.460049 3798194 disk_worker.go:509] system disk:vda1 I1121 22:12:57.460090 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:12:58.446530 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:12:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:12:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:12:59.450133 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:12:59.471674 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:13:00.066028 3798194 disk_info.go:125] begin check local disk info of client I1121 22:13:00.069846 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:13:00.069855 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00049abc0 0xc00049ac00] E1121 22:13:06.411975 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:13:06.411997 3798194 memory.go:184] no items to output this cycle I1121 22:13:06.412159 3798194 cpu.go:275] no items to output this cycle E1121 22:13:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:13:16.411981 3798194 memory.go:184] no items to output this cycle I1121 22:13:16.412140 3798194 cpu.go:275] no items to output this cycle E1121 22:13:26.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:13:26.413266 3798194 memory.go:191] Add success. I1121 22:13:26.417763 3798194 cpu.go:282] Add success. I1121 22:13:26.441412 3798194 net.go:642] Add success. I1121 22:13:26.443107 3798194 net.go:778] primary dev: ETH0 I1121 22:13:26.443136 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:13:26.443149 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:13:29.450477 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:13:29.450568 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:13:29.450598 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:13:36.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:13:36.411972 3798194 memory.go:184] no items to output this cycle I1121 22:13:36.412129 3798194 cpu.go:275] no items to output this cycle E1121 22:13:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:13:46.411971 3798194 memory.go:184] no items to output this cycle I1121 22:13:46.412126 3798194 cpu.go:275] no items to output this cycle E1121 22:13:56.411931 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:13:56.411965 3798194 memory.go:191] Add success. W1121 22:13:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:13:56.412010 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:13:56.412013 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:13:56.412358 3798194 cpu.go:282] Add success. I1121 22:13:56.440972 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:13:56.441552 3798194 net.go:642] Add success. I1121 22:13:56.445394 3798194 net.go:778] primary dev: ETH0 I1121 22:13:56.445412 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:13:56.445424 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:13:57.450593 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:13:57.450799 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:13:57.450810 3798194 disk_worker.go:723] disk space is not compliant W1121 22:13:57.450813 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:13:57.459336 3798194 disk_worker.go:509] system disk:vda1 I1121 22:13:57.459378 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:13:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:13:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:13:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:13:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:13:59.471656 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:14:00.074163 3798194 disk_info.go:125] begin check local disk info of client I1121 22:14:00.080035 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:14:00.080050 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c5f40 0xc0003fe000] E1121 22:14:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:14:06.411983 3798194 memory.go:184] no items to output this cycle I1121 22:14:06.412137 3798194 cpu.go:275] no items to output this cycle E1121 22:14:16.411972 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:14:16.411996 3798194 memory.go:184] no items to output this cycle I1121 22:14:16.412292 3798194 cpu.go:275] no items to output this cycle E1121 22:14:26.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:14:26.412271 3798194 memory.go:191] Add success. I1121 22:14:26.416735 3798194 cpu.go:282] Add success. I1121 22:14:26.441427 3798194 net.go:642] Add success. I1121 22:14:26.442926 3798194 net.go:778] primary dev: ETH0 I1121 22:14:26.442940 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:14:26.442953 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:14:29.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:14:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:14:29.450498 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:14:36.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:14:36.413249 3798194 memory.go:184] no items to output this cycle I1121 22:14:36.413399 3798194 cpu.go:275] no items to output this cycle E1121 22:14:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:14:46.412256 3798194 memory.go:184] no items to output this cycle I1121 22:14:46.412404 3798194 cpu.go:275] no items to output this cycle E1121 22:14:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:14:56.411946 3798194 memory.go:191] Add success. W1121 22:14:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:14:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:14:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:14:56.412336 3798194 cpu.go:282] Add success. I1121 22:14:56.441911 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:14:56.442501 3798194 net.go:642] Add success. I1121 22:14:56.446389 3798194 net.go:778] primary dev: ETH0 I1121 22:14:56.446404 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:14:56.446418 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:14:57.450592 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:14:57.450762 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:14:57.450815 3798194 disk_worker.go:723] disk space is not compliant W1121 22:14:57.450818 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:14:57.460993 3798194 disk_worker.go:509] system disk:vda1 I1121 22:14:57.461035 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:14:58.446385 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:14:59.450309 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:14:59.450392 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:14:59.450417 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:14:59.470947 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:15:00.082155 3798194 disk_info.go:125] begin check local disk info of client I1121 22:15:00.085976 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:15:00.085985 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002ffc40 0xc0002ffc80] E1121 22:15:06.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:15:06.413264 3798194 memory.go:184] no items to output this cycle I1121 22:15:06.413405 3798194 cpu.go:275] no items to output this cycle E1121 22:15:16.412328 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:15:16.412350 3798194 memory.go:184] no items to output this cycle I1121 22:15:16.412495 3798194 cpu.go:275] no items to output this cycle I1121 22:15:17.567649 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:15:17.567658 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:15:26.411332 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:15:26.411364 3798194 memory.go:191] Add success. I1121 22:15:26.411698 3798194 cpu.go:282] Add success. I1121 22:15:26.441256 3798194 net.go:642] Add success. I1121 22:15:26.445356 3798194 net.go:778] primary dev: ETH0 I1121 22:15:26.445371 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:15:26.445385 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:15:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:15:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:15:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:15:36.412331 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:15:36.412352 3798194 memory.go:184] no items to output this cycle I1121 22:15:36.412495 3798194 cpu.go:275] no items to output this cycle E1121 22:15:46.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:15:46.413256 3798194 memory.go:184] no items to output this cycle I1121 22:15:46.413413 3798194 cpu.go:275] no items to output this cycle E1121 22:15:56.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:15:56.413271 3798194 memory.go:191] Add success. W1121 22:15:56.413302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:15:56.413313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:15:56.413317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:15:56.417310 3798194 cpu.go:282] Add success. I1121 22:15:56.444230 3798194 event_worker.go:152] Polling the log file for events... I1121 22:15:56.444447 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:15:56.444931 3798194 net.go:642] Add success. I1121 22:15:56.446506 3798194 net.go:778] primary dev: ETH0 I1121 22:15:56.446520 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:15:56.446533 3798194 net.go:659] Add eni metrics success, count: 7 E1121 22:15:57.450945 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 22:15:57.450957 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:15:57.451184 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:15:57.451195 3798194 disk_worker.go:723] disk space is not compliant W1121 22:15:57.451198 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:15:57.459930 3798194 disk_worker.go:509] system disk:vda1 I1121 22:15:57.459970 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 22:15:58.448460 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 22:15:58.448477 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 22:15:59.448982 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 22:15:59.450057 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:15:59.450118 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:15:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:15:59.471688 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:16:00.086088 3798194 disk_info.go:125] begin check local disk info of client I1121 22:16:00.089963 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:16:00.089972 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e1700 0xc0002e1740] E1121 22:16:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:16:06.411986 3798194 memory.go:184] no items to output this cycle I1121 22:16:06.412143 3798194 cpu.go:275] no items to output this cycle E1121 22:16:16.412787 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:16:16.412810 3798194 memory.go:184] no items to output this cycle I1121 22:16:16.412959 3798194 cpu.go:275] no items to output this cycle E1121 22:16:26.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:16:26.412889 3798194 memory.go:191] Add success. I1121 22:16:26.417600 3798194 cpu.go:282] Add success. I1121 22:16:26.441418 3798194 net.go:642] Add success. I1121 22:16:26.443472 3798194 net.go:778] primary dev: ETH0 I1121 22:16:26.443487 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:16:26.443500 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:16:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:16:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:16:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:16:36.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:16:36.413252 3798194 memory.go:184] no items to output this cycle I1121 22:16:36.413396 3798194 cpu.go:275] no items to output this cycle E1121 22:16:46.413224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:16:46.413246 3798194 memory.go:184] no items to output this cycle I1121 22:16:46.413388 3798194 cpu.go:275] no items to output this cycle E1121 22:16:56.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:16:56.412278 3798194 memory.go:191] Add success. W1121 22:16:56.412310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:16:56.412321 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:16:56.412325 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:16:56.417015 3798194 cpu.go:282] Add success. I1121 22:16:56.441417 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:16:56.441894 3798194 net.go:642] Add success. I1121 22:16:56.443488 3798194 net.go:778] primary dev: ETH0 I1121 22:16:56.443504 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:16:56.443517 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:16:57.450765 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:16:57.450922 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:16:57.450976 3798194 disk_worker.go:723] disk space is not compliant W1121 22:16:57.450980 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:16:57.459737 3798194 disk_worker.go:509] system disk:vda1 I1121 22:16:57.459896 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:16:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:16:59.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:16:59.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:16:59.450499 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:16:59.471015 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:17:00.090076 3798194 disk_info.go:125] begin check local disk info of client I1121 22:17:00.093904 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:17:00.093914 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000468a80 0xc000468ac0] E1121 22:17:06.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:17:06.413262 3798194 memory.go:184] no items to output this cycle I1121 22:17:06.413411 3798194 cpu.go:275] no items to output this cycle E1121 22:17:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:17:16.413261 3798194 memory.go:184] no items to output this cycle I1121 22:17:16.413405 3798194 cpu.go:275] no items to output this cycle E1121 22:17:26.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:17:26.412274 3798194 memory.go:191] Add success. I1121 22:17:26.416570 3798194 cpu.go:282] Add success. I1121 22:17:26.442139 3798194 net.go:642] Add success. I1121 22:17:26.443828 3798194 net.go:778] primary dev: ETH0 I1121 22:17:26.443843 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:17:26.443855 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:17:29.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:17:29.450489 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:17:29.450522 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:17:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:17:36.413258 3798194 memory.go:184] no items to output this cycle I1121 22:17:36.413400 3798194 cpu.go:275] no items to output this cycle E1121 22:17:46.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:17:46.412253 3798194 memory.go:184] no items to output this cycle I1121 22:17:46.412400 3798194 cpu.go:275] no items to output this cycle E1121 22:17:56.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:17:56.412261 3798194 memory.go:191] Add success. W1121 22:17:56.412293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:17:56.412304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:17:56.412307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:17:56.416760 3798194 cpu.go:282] Add success. I1121 22:17:56.441175 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:17:56.441682 3798194 net.go:642] Add success. I1121 22:17:56.443324 3798194 net.go:778] primary dev: ETH0 I1121 22:17:56.443339 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:17:56.443351 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:17:57.450572 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:17:57.450741 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:17:57.450796 3798194 disk_worker.go:723] disk space is not compliant W1121 22:17:57.450799 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:17:57.459214 3798194 disk_worker.go:509] system disk:vda1 I1121 22:17:57.459256 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:17:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:17:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:17:59.450119 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:17:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:17:59.471675 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:18:00.096805 3798194 disk_info.go:125] begin check local disk info of client I1121 22:18:00.100682 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:18:00.100691 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e4200 0xc0003e4240] E1121 22:18:06.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:18:06.412251 3798194 memory.go:184] no items to output this cycle I1121 22:18:06.412393 3798194 cpu.go:275] no items to output this cycle E1121 22:18:16.412339 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:18:16.412363 3798194 memory.go:184] no items to output this cycle I1121 22:18:16.412512 3798194 cpu.go:275] no items to output this cycle I1121 22:18:17.567814 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:18:17.567822 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:18:26.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:18:26.412264 3798194 memory.go:191] Add success. I1121 22:18:26.416932 3798194 cpu.go:282] Add success. I1121 22:18:26.441480 3798194 net.go:642] Add success. I1121 22:18:26.442972 3798194 net.go:778] primary dev: ETH0 I1121 22:18:26.442986 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:18:26.443001 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:18:29.450043 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:18:29.450132 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:18:29.450161 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:18:36.413249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:18:36.413272 3798194 memory.go:184] no items to output this cycle I1121 22:18:36.413409 3798194 cpu.go:275] no items to output this cycle E1121 22:18:46.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:18:46.412263 3798194 memory.go:184] no items to output this cycle I1121 22:18:46.412405 3798194 cpu.go:275] no items to output this cycle E1121 22:18:56.412256 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:18:56.412291 3798194 memory.go:191] Add success. W1121 22:18:56.412317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:18:56.412328 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:18:56.412332 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:18:56.417314 3798194 cpu.go:282] Add success. I1121 22:18:56.441676 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:18:56.442170 3798194 net.go:642] Add success. I1121 22:18:56.443808 3798194 net.go:778] primary dev: ETH0 I1121 22:18:56.443822 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:18:56.443834 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:18:57.451008 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:18:57.451231 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:18:57.451241 3798194 disk_worker.go:723] disk space is not compliant W1121 22:18:57.451244 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:18:57.459959 3798194 disk_worker.go:509] system disk:vda1 I1121 22:18:57.460003 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:18:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:18:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:18:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:18:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:18:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:19:00.103150 3798194 disk_info.go:125] begin check local disk info of client I1121 22:19:00.106987 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:19:00.106996 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e5180 0xc0003e51c0] E1121 22:19:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:19:06.411983 3798194 memory.go:184] no items to output this cycle I1121 22:19:06.412140 3798194 cpu.go:275] no items to output this cycle E1121 22:19:16.412865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:19:16.412887 3798194 memory.go:184] no items to output this cycle I1121 22:19:16.413032 3798194 cpu.go:275] no items to output this cycle E1121 22:19:26.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:19:26.411947 3798194 memory.go:191] Add success. I1121 22:19:26.412276 3798194 cpu.go:282] Add success. I1121 22:19:26.441763 3798194 net.go:642] Add success. I1121 22:19:26.446348 3798194 net.go:778] primary dev: ETH0 I1121 22:19:26.446364 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:19:26.446378 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:19:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:19:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:19:29.450129 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:19:36.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:19:36.413254 3798194 memory.go:184] no items to output this cycle I1121 22:19:36.413393 3798194 cpu.go:275] no items to output this cycle E1121 22:19:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:19:46.412258 3798194 memory.go:184] no items to output this cycle I1121 22:19:46.412413 3798194 cpu.go:275] no items to output this cycle E1121 22:19:56.412305 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:19:56.412339 3798194 memory.go:191] Add success. W1121 22:19:56.412367 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:19:56.412379 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:19:56.412382 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:19:56.416754 3798194 cpu.go:282] Add success. I1121 22:19:56.441154 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:19:56.441627 3798194 net.go:642] Add success. I1121 22:19:56.450583 3798194 net.go:778] primary dev: ETH0 I1121 22:19:56.450599 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:19:56.450612 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:19:57.446405 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:19:57.446919 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:19:57.446989 3798194 disk_worker.go:723] disk space is not compliant W1121 22:19:57.446997 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:19:57.450361 3798194 disk_worker.go:509] system disk:vda1 I1121 22:19:57.450399 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:19:58.446372 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:19:59.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:19:59.450467 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:19:59.450491 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:19:59.471005 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:20:00.109145 3798194 disk_info.go:125] begin check local disk info of client I1121 22:20:00.112941 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:20:00.112950 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a5a80 0xc0004a5ac0] E1121 22:20:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:20:06.411987 3798194 memory.go:184] no items to output this cycle I1121 22:20:06.412149 3798194 cpu.go:275] no items to output this cycle E1121 22:20:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:20:16.411987 3798194 memory.go:184] no items to output this cycle I1121 22:20:16.412159 3798194 cpu.go:275] no items to output this cycle E1121 22:20:26.413082 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:20:26.413128 3798194 memory.go:191] Add success. I1121 22:20:26.417532 3798194 cpu.go:282] Add success. I1121 22:20:26.442264 3798194 net.go:642] Add success. I1121 22:20:26.443921 3798194 net.go:778] primary dev: ETH0 I1121 22:20:26.443939 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:20:26.443952 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:20:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:20:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:20:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:20:36.411992 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:20:36.412016 3798194 memory.go:184] no items to output this cycle I1121 22:20:36.412177 3798194 cpu.go:275] no items to output this cycle E1121 22:20:46.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:20:46.412885 3798194 memory.go:184] no items to output this cycle I1121 22:20:46.413035 3798194 cpu.go:275] no items to output this cycle E1121 22:20:56.413123 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:20:56.413156 3798194 memory.go:191] Add success. W1121 22:20:56.413189 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:20:56.413201 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:20:56.413204 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:20:56.417674 3798194 cpu.go:282] Add success. I1121 22:20:56.444271 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:20:56.444750 3798194 net.go:642] Add success. I1121 22:20:56.446402 3798194 net.go:778] primary dev: ETH0 I1121 22:20:56.446416 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:20:56.446429 3798194 net.go:659] Add eni metrics success, count: 7 E1121 22:20:57.450651 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 22:20:57.450666 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:20:57.450869 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:20:57.450879 3798194 disk_worker.go:723] disk space is not compliant W1121 22:20:57.450882 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:20:57.460225 3798194 disk_worker.go:509] system disk:vda1 I1121 22:20:57.460267 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 22:20:58.448083 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 22:20:58.448121 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 22:20:59.448786 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 22:20:59.449863 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:20:59.449914 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:20:59.449934 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:20:59.471494 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:21:00.115140 3798194 disk_info.go:125] begin check local disk info of client I1121 22:21:00.118942 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:21:00.118951 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000326e80 0xc000326ec0] E1121 22:21:06.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:21:06.413251 3798194 memory.go:184] no items to output this cycle I1121 22:21:06.413396 3798194 cpu.go:275] no items to output this cycle E1121 22:21:16.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:21:16.413253 3798194 memory.go:184] no items to output this cycle I1121 22:21:16.413412 3798194 cpu.go:275] no items to output this cycle I1121 22:21:17.568217 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:21:17.568225 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:21:26.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:21:26.413282 3798194 memory.go:191] Add success. I1121 22:21:26.417654 3798194 cpu.go:282] Add success. I1121 22:21:26.442527 3798194 net.go:642] Add success. I1121 22:21:26.444045 3798194 net.go:778] primary dev: ETH0 I1121 22:21:26.444060 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:21:26.444073 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:21:29.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:21:29.450496 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:21:29.450528 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:21:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:21:36.412254 3798194 memory.go:184] no items to output this cycle I1121 22:21:36.412398 3798194 cpu.go:275] no items to output this cycle E1121 22:21:46.412785 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:21:46.412808 3798194 memory.go:184] no items to output this cycle I1121 22:21:46.412955 3798194 cpu.go:275] no items to output this cycle E1121 22:21:56.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:21:56.412257 3798194 memory.go:191] Add success. W1121 22:21:56.412285 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:21:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:21:56.412300 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:21:56.416747 3798194 cpu.go:282] Add success. I1121 22:21:56.441164 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:21:56.441637 3798194 net.go:642] Add success. I1121 22:21:56.443182 3798194 net.go:778] primary dev: ETH0 I1121 22:21:56.443197 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:21:56.443209 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:21:57.450366 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:21:57.450530 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:21:57.450595 3798194 disk_worker.go:723] disk space is not compliant W1121 22:21:57.450599 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:21:57.458990 3798194 disk_worker.go:509] system disk:vda1 I1121 22:21:57.459030 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:21:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:21:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:21:59.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:21:59.450132 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:21:59.471664 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:22:00.119052 3798194 disk_info.go:125] begin check local disk info of client I1121 22:22:00.122957 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:22:00.122969 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000313400 0xc000313440] E1121 22:22:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:22:06.411981 3798194 memory.go:184] no items to output this cycle I1121 22:22:06.412143 3798194 cpu.go:275] no items to output this cycle E1121 22:22:16.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:22:16.411984 3798194 memory.go:184] no items to output this cycle I1121 22:22:16.412146 3798194 cpu.go:275] no items to output this cycle E1121 22:22:26.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:22:26.413260 3798194 memory.go:191] Add success. I1121 22:22:26.417675 3798194 cpu.go:282] Add success. I1121 22:22:26.442196 3798194 net.go:642] Add success. I1121 22:22:26.443724 3798194 net.go:778] primary dev: ETH0 I1121 22:22:26.443739 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:22:26.443754 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:22:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:22:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:22:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:22:36.413182 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:22:36.413206 3798194 memory.go:184] no items to output this cycle I1121 22:22:36.413355 3798194 cpu.go:275] no items to output this cycle E1121 22:22:46.411976 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:22:46.412017 3798194 memory.go:184] no items to output this cycle I1121 22:22:46.412195 3798194 cpu.go:275] no items to output this cycle E1121 22:22:56.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:22:56.412288 3798194 memory.go:191] Add success. W1121 22:22:56.412323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:22:56.412335 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:22:56.412338 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:22:56.416836 3798194 cpu.go:282] Add success. I1121 22:22:56.441275 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:22:56.441763 3798194 net.go:642] Add success. I1121 22:22:56.443223 3798194 net.go:778] primary dev: ETH0 I1121 22:22:56.443237 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:22:56.443249 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:22:57.450961 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:22:57.451214 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:22:57.451225 3798194 disk_worker.go:723] disk space is not compliant W1121 22:22:57.451228 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:22:57.459988 3798194 disk_worker.go:509] system disk:vda1 I1121 22:22:57.460073 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:22:58.446663 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:22:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:22:59.450487 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:22:59.450513 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:22:59.471036 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:23:00.123071 3798194 disk_info.go:125] begin check local disk info of client I1121 22:23:00.126896 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:23:00.126907 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0d40 0xc0002e0d80] E1121 22:23:06.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:23:06.411973 3798194 memory.go:184] no items to output this cycle I1121 22:23:06.412132 3798194 cpu.go:275] no items to output this cycle E1121 22:23:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:23:16.411977 3798194 memory.go:184] no items to output this cycle I1121 22:23:16.412135 3798194 cpu.go:275] no items to output this cycle E1121 22:23:26.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:23:26.413259 3798194 memory.go:191] Add success. I1121 22:23:26.417608 3798194 cpu.go:282] Add success. I1121 22:23:26.442108 3798194 net.go:642] Add success. I1121 22:23:26.443828 3798194 net.go:778] primary dev: ETH0 I1121 22:23:26.443842 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:23:26.443855 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:23:29.449860 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:23:29.449937 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:23:29.449965 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:23:36.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:23:36.413261 3798194 memory.go:184] no items to output this cycle I1121 22:23:36.413402 3798194 cpu.go:275] no items to output this cycle E1121 22:23:46.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:23:46.412264 3798194 memory.go:184] no items to output this cycle I1121 22:23:46.412410 3798194 cpu.go:275] no items to output this cycle E1121 22:23:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:23:56.413268 3798194 memory.go:191] Add success. W1121 22:23:56.413296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:23:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:23:56.413311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:23:56.417616 3798194 cpu.go:282] Add success. I1121 22:23:56.440945 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:23:56.441538 3798194 net.go:642] Add success. I1121 22:23:56.443089 3798194 net.go:778] primary dev: ETH0 I1121 22:23:56.443103 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:23:56.443131 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:23:57.450933 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:23:57.451150 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:23:57.451161 3798194 disk_worker.go:723] disk space is not compliant W1121 22:23:57.451164 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:23:57.459874 3798194 disk_worker.go:509] system disk:vda1 I1121 22:23:57.459915 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:23:58.445997 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:23:59.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:23:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:23:59.450132 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:23:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:24:00.129150 3798194 disk_info.go:125] begin check local disk info of client I1121 22:24:00.132573 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:24:00.132581 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000452880 0xc0004528c0] E1121 22:24:06.412201 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:24:06.412224 3798194 memory.go:184] no items to output this cycle I1121 22:24:06.412374 3798194 cpu.go:275] no items to output this cycle E1121 22:24:16.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:24:16.412268 3798194 memory.go:184] no items to output this cycle I1121 22:24:16.412426 3798194 cpu.go:275] no items to output this cycle I1121 22:24:17.568387 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:24:17.568397 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:24:26.412512 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:24:26.412546 3798194 memory.go:191] Add success. I1121 22:24:26.416887 3798194 cpu.go:282] Add success. I1121 22:24:26.441443 3798194 net.go:642] Add success. I1121 22:24:26.442952 3798194 net.go:778] primary dev: ETH0 I1121 22:24:26.442967 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:24:26.442982 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:24:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:24:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:24:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:24:36.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:24:36.411984 3798194 memory.go:184] no items to output this cycle I1121 22:24:36.412145 3798194 cpu.go:275] no items to output this cycle E1121 22:24:46.412791 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:24:46.412814 3798194 memory.go:184] no items to output this cycle I1121 22:24:46.412960 3798194 cpu.go:275] no items to output this cycle E1121 22:24:56.412937 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:24:56.412978 3798194 memory.go:191] Add success. W1121 22:24:56.413012 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:24:56.413024 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:24:56.413027 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:24:56.417423 3798194 cpu.go:282] Add success. I1121 22:24:56.441794 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:24:56.442384 3798194 net.go:642] Add success. I1121 22:24:56.443883 3798194 net.go:778] primary dev: ETH0 I1121 22:24:56.443900 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:24:56.443913 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:24:57.451062 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:24:57.451250 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:24:57.451307 3798194 disk_worker.go:723] disk space is not compliant W1121 22:24:57.451310 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:24:57.460094 3798194 disk_worker.go:509] system disk:vda1 I1121 22:24:57.460155 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:24:58.446374 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:24:59.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:24:59.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:24:59.450489 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:24:59.471095 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:25:00.132671 3798194 disk_info.go:125] begin check local disk info of client I1121 22:25:00.136497 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:25:00.136506 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b81c0 0xc0003b8200] E1121 22:25:06.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:25:06.411978 3798194 memory.go:184] no items to output this cycle I1121 22:25:06.412135 3798194 cpu.go:275] no items to output this cycle E1121 22:25:16.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:25:16.413252 3798194 memory.go:184] no items to output this cycle I1121 22:25:16.413397 3798194 cpu.go:275] no items to output this cycle E1121 22:25:26.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:25:26.413275 3798194 memory.go:191] Add success. I1121 22:25:26.417520 3798194 cpu.go:282] Add success. I1121 22:25:26.442320 3798194 net.go:642] Add success. I1121 22:25:26.443897 3798194 net.go:778] primary dev: ETH0 I1121 22:25:26.443912 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:25:26.443926 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:25:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:25:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:25:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:25:36.411945 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:25:36.411969 3798194 memory.go:184] no items to output this cycle I1121 22:25:36.412107 3798194 cpu.go:275] no items to output this cycle E1121 22:25:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:25:46.411988 3798194 memory.go:184] no items to output this cycle I1121 22:25:46.412150 3798194 cpu.go:275] no items to output this cycle E1121 22:25:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:25:56.411948 3798194 memory.go:191] Add success. W1121 22:25:56.411975 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:25:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:25:56.411989 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:25:56.412343 3798194 cpu.go:282] Add success. I1121 22:25:56.441568 3798194 event_worker.go:152] Polling the log file for events... I1121 22:25:56.444684 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:25:56.445198 3798194 net.go:642] Add success. I1121 22:25:56.449501 3798194 net.go:778] primary dev: ETH0 I1121 22:25:56.449515 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:25:56.449527 3798194 net.go:659] Add eni metrics success, count: 7 E1121 22:25:57.450696 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 22:25:57.450711 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:25:57.450901 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:25:57.450910 3798194 disk_worker.go:723] disk space is not compliant W1121 22:25:57.450913 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:25:57.459761 3798194 disk_worker.go:509] system disk:vda1 I1121 22:25:57.459802 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 22:25:58.448459 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 22:25:58.448475 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 22:25:59.449160 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 22:25:59.450234 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:25:59.450284 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:25:59.450304 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:25:59.470728 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:26:00.138142 3798194 disk_info.go:125] begin check local disk info of client I1121 22:26:00.141940 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:26:00.141952 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00048f380 0xc00048f3c0] E1121 22:26:06.412763 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:26:06.412795 3798194 memory.go:184] no items to output this cycle I1121 22:26:06.412950 3798194 cpu.go:275] no items to output this cycle E1121 22:26:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:26:16.412262 3798194 memory.go:184] no items to output this cycle I1121 22:26:16.412404 3798194 cpu.go:275] no items to output this cycle E1121 22:26:26.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:26:26.413280 3798194 memory.go:191] Add success. I1121 22:26:26.417850 3798194 cpu.go:282] Add success. I1121 22:26:26.441290 3798194 net.go:642] Add success. I1121 22:26:26.442825 3798194 net.go:778] primary dev: ETH0 I1121 22:26:26.442840 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:26:26.442852 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:26:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:26:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:26:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:26:36.412926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:26:36.412949 3798194 memory.go:184] no items to output this cycle I1121 22:26:36.413100 3798194 cpu.go:275] no items to output this cycle E1121 22:26:46.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:26:46.413263 3798194 memory.go:184] no items to output this cycle I1121 22:26:46.413420 3798194 cpu.go:275] no items to output this cycle E1121 22:26:56.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:26:56.413274 3798194 memory.go:191] Add success. W1121 22:26:56.413302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:26:56.413313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:26:56.413316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:26:56.421331 3798194 cpu.go:282] Add success. I1121 22:26:56.440955 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:26:56.441554 3798194 net.go:642] Add success. I1121 22:26:56.443031 3798194 net.go:778] primary dev: ETH0 I1121 22:26:56.443045 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:26:56.443058 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:26:57.450874 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:26:57.451069 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:26:57.451080 3798194 disk_worker.go:723] disk space is not compliant W1121 22:26:57.451084 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:26:57.459891 3798194 disk_worker.go:509] system disk:vda1 I1121 22:26:57.459935 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:26:58.446548 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:26:59.450648 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:26:59.450727 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:26:59.450752 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:26:59.471256 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:27:00.144145 3798194 disk_info.go:125] begin check local disk info of client I1121 22:27:00.147987 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:27:00.147996 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e740 0xc00030e780] E1121 22:27:06.413250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:27:06.413273 3798194 memory.go:184] no items to output this cycle I1121 22:27:06.413416 3798194 cpu.go:275] no items to output this cycle E1121 22:27:16.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:27:16.413256 3798194 memory.go:184] no items to output this cycle I1121 22:27:16.413406 3798194 cpu.go:275] no items to output this cycle I1121 22:27:17.568560 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:27:17.568568 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:27:26.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:27:26.413278 3798194 memory.go:191] Add success. I1121 22:27:26.417522 3798194 cpu.go:282] Add success. I1121 22:27:26.442092 3798194 net.go:642] Add success. I1121 22:27:26.443741 3798194 net.go:778] primary dev: ETH0 I1121 22:27:26.443756 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:27:26.443768 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:27:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:27:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:27:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:27:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:27:36.412254 3798194 memory.go:184] no items to output this cycle I1121 22:27:36.412400 3798194 cpu.go:275] no items to output this cycle E1121 22:27:46.412671 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:27:46.412696 3798194 memory.go:184] no items to output this cycle I1121 22:27:46.412848 3798194 cpu.go:275] no items to output this cycle E1121 22:27:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:27:56.412275 3798194 memory.go:191] Add success. W1121 22:27:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:27:56.412314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:27:56.412319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:27:56.416474 3798194 cpu.go:282] Add success. I1121 22:27:56.440936 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:27:56.441544 3798194 net.go:642] Add success. I1121 22:27:56.443181 3798194 net.go:778] primary dev: ETH0 I1121 22:27:56.443195 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:27:56.443208 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:27:57.449147 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:27:57.449395 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:27:57.449407 3798194 disk_worker.go:723] disk space is not compliant W1121 22:27:57.449410 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:27:57.451261 3798194 disk_worker.go:509] system disk:vda1 I1121 22:27:57.451308 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:27:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:27:59.450382 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:27:59.450465 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:27:59.450491 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:27:59.471012 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:28:00.150138 3798194 disk_info.go:125] begin check local disk info of client I1121 22:28:00.154054 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:28:00.154063 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041c9c0 0xc00041ca00] E1121 22:28:06.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:28:06.412252 3798194 memory.go:184] no items to output this cycle I1121 22:28:06.412394 3798194 cpu.go:275] no items to output this cycle E1121 22:28:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:28:16.413260 3798194 memory.go:184] no items to output this cycle I1121 22:28:16.413411 3798194 cpu.go:275] no items to output this cycle E1121 22:28:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:28:26.411957 3798194 memory.go:191] Add success. I1121 22:28:26.412337 3798194 cpu.go:282] Add success. I1121 22:28:26.441949 3798194 net.go:642] Add success. I1121 22:28:26.445811 3798194 net.go:778] primary dev: ETH0 I1121 22:28:26.445826 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:28:26.445839 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:28:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:28:29.450088 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:28:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:28:36.413162 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:28:36.413185 3798194 memory.go:184] no items to output this cycle I1121 22:28:36.413330 3798194 cpu.go:275] no items to output this cycle E1121 22:28:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:28:46.411984 3798194 memory.go:184] no items to output this cycle I1121 22:28:46.412145 3798194 cpu.go:275] no items to output this cycle E1121 22:28:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:28:56.411956 3798194 memory.go:191] Add success. W1121 22:28:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:28:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:28:56.412006 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:28:56.412366 3798194 cpu.go:282] Add success. I1121 22:28:56.441811 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:28:56.442310 3798194 net.go:642] Add success. I1121 22:28:56.446351 3798194 net.go:778] primary dev: ETH0 I1121 22:28:56.446366 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:28:56.446380 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:28:57.451134 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:28:57.451348 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:28:57.451360 3798194 disk_worker.go:723] disk space is not compliant W1121 22:28:57.451363 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:28:57.463048 3798194 disk_worker.go:509] system disk:vda1 I1121 22:28:57.463088 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:28:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:28:59.450008 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:28:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:28:59.450144 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:28:59.470680 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:29:00.154166 3798194 disk_info.go:125] begin check local disk info of client I1121 22:29:00.157993 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:29:00.158002 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000372340 0xc000372380] E1121 22:29:06.413175 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:29:06.413197 3798194 memory.go:184] no items to output this cycle I1121 22:29:06.413344 3798194 cpu.go:275] no items to output this cycle E1121 22:29:16.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:29:16.413263 3798194 memory.go:184] no items to output this cycle I1121 22:29:16.413411 3798194 cpu.go:275] no items to output this cycle E1121 22:29:26.411933 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:29:26.411966 3798194 memory.go:191] Add success. I1121 22:29:26.412293 3798194 cpu.go:282] Add success. I1121 22:29:26.441771 3798194 net.go:642] Add success. I1121 22:29:26.445797 3798194 net.go:778] primary dev: ETH0 I1121 22:29:26.445813 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:29:26.445827 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:29:29.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:29:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:29:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:29:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:29:36.412259 3798194 memory.go:184] no items to output this cycle I1121 22:29:36.412413 3798194 cpu.go:275] no items to output this cycle E1121 22:29:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:29:46.411989 3798194 memory.go:184] no items to output this cycle I1121 22:29:46.412154 3798194 cpu.go:275] no items to output this cycle E1121 22:29:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:29:56.411947 3798194 memory.go:191] Add success. W1121 22:29:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:29:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:29:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:29:56.412387 3798194 cpu.go:282] Add success. I1121 22:29:56.441870 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:29:56.442476 3798194 net.go:642] Add success. I1121 22:29:56.446416 3798194 net.go:778] primary dev: ETH0 I1121 22:29:56.446431 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:29:56.446445 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:29:57.451828 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:29:57.452038 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:29:57.452062 3798194 disk_worker.go:723] disk space is not compliant W1121 22:29:57.452066 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:29:57.462263 3798194 disk_worker.go:509] system disk:vda1 I1121 22:29:57.462304 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:29:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:29:59.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:29:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:29:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:29:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:30:00.158107 3798194 disk_info.go:125] begin check local disk info of client I1121 22:30:00.161939 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:30:00.161948 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000426540 0xc000426580] E1121 22:30:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:30:06.412259 3798194 memory.go:184] no items to output this cycle I1121 22:30:06.412404 3798194 cpu.go:275] no items to output this cycle E1121 22:30:16.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:30:16.411989 3798194 memory.go:184] no items to output this cycle I1121 22:30:16.412148 3798194 cpu.go:275] no items to output this cycle I1121 22:30:17.568735 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:30:17.568744 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:30:26.411309 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:30:26.411352 3798194 memory.go:191] Add success. I1121 22:30:26.411674 3798194 cpu.go:282] Add success. I1121 22:30:26.441259 3798194 net.go:642] Add success. I1121 22:30:26.445441 3798194 net.go:778] primary dev: ETH0 I1121 22:30:26.445458 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:30:26.445474 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:30:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:30:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:30:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:30:36.412759 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:30:36.412783 3798194 memory.go:184] no items to output this cycle I1121 22:30:36.412934 3798194 cpu.go:275] no items to output this cycle E1121 22:30:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:30:46.411986 3798194 memory.go:184] no items to output this cycle I1121 22:30:46.412156 3798194 cpu.go:275] no items to output this cycle E1121 22:30:56.411907 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:30:56.411938 3798194 memory.go:191] Add success. W1121 22:30:56.411970 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:30:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:30:56.411985 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:30:56.412331 3798194 cpu.go:282] Add success. I1121 22:30:56.447212 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:30:56.447716 3798194 net.go:642] Add success. I1121 22:30:56.451731 3798194 net.go:778] primary dev: ETH0 I1121 22:30:56.451746 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:30:56.451758 3798194 net.go:659] Add eni metrics success, count: 7 E1121 22:30:57.450647 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 22:30:57.450662 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:30:57.450822 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:30:57.450877 3798194 disk_worker.go:723] disk space is not compliant W1121 22:30:57.450880 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:30:57.459362 3798194 disk_worker.go:509] system disk:vda1 I1121 22:30:57.459418 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 22:30:58.448095 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 22:30:58.448127 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 22:30:59.448600 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 22:30:59.449674 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:30:59.449726 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:30:59.449745 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:30:59.471252 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:31:00.162051 3798194 disk_info.go:125] begin check local disk info of client I1121 22:31:00.165892 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:31:00.165901 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00048f6c0 0xc00048f700] E1121 22:31:06.413160 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:31:06.413183 3798194 memory.go:184] no items to output this cycle I1121 22:31:06.413331 3798194 cpu.go:275] no items to output this cycle E1121 22:31:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:31:16.412257 3798194 memory.go:184] no items to output this cycle I1121 22:31:16.412409 3798194 cpu.go:275] no items to output this cycle E1121 22:31:26.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:31:26.412893 3798194 memory.go:191] Add success. I1121 22:31:26.417022 3798194 cpu.go:282] Add success. I1121 22:31:26.441444 3798194 net.go:642] Add success. I1121 22:31:26.442969 3798194 net.go:778] primary dev: ETH0 I1121 22:31:26.442983 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:31:26.442997 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:31:29.449943 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:31:29.450017 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:31:29.450042 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:31:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:31:36.413265 3798194 memory.go:184] no items to output this cycle I1121 22:31:36.413409 3798194 cpu.go:275] no items to output this cycle E1121 22:31:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:31:46.411978 3798194 memory.go:184] no items to output this cycle I1121 22:31:46.412146 3798194 cpu.go:275] no items to output this cycle E1121 22:31:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:31:56.413269 3798194 memory.go:191] Add success. W1121 22:31:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:31:56.413320 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:31:56.413323 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:31:56.417301 3798194 cpu.go:282] Add success. I1121 22:31:56.441715 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:31:56.442221 3798194 net.go:642] Add success. I1121 22:31:56.443947 3798194 net.go:778] primary dev: ETH0 I1121 22:31:56.443961 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:31:56.443982 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:31:57.453071 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:31:57.453299 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:31:57.453311 3798194 disk_worker.go:723] disk space is not compliant W1121 22:31:57.453314 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:31:57.464581 3798194 disk_worker.go:509] system disk:vda1 I1121 22:31:57.464625 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:31:58.446429 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:31:59.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:31:59.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:31:59.450496 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:31:59.471015 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:32:00.165998 3798194 disk_info.go:125] begin check local disk info of client I1121 22:32:00.169825 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:32:00.169834 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e4300 0xc0003e4340] E1121 22:32:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:32:06.412253 3798194 memory.go:184] no items to output this cycle I1121 22:32:06.412405 3798194 cpu.go:275] no items to output this cycle E1121 22:32:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:32:16.411978 3798194 memory.go:184] no items to output this cycle I1121 22:32:16.412136 3798194 cpu.go:275] no items to output this cycle E1121 22:32:26.412842 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:32:26.412877 3798194 memory.go:191] Add success. I1121 22:32:26.417313 3798194 cpu.go:282] Add success. I1121 22:32:26.441736 3798194 net.go:642] Add success. I1121 22:32:26.443400 3798194 net.go:778] primary dev: ETH0 I1121 22:32:26.443415 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:32:26.443535 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:32:29.450133 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:32:29.450213 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:32:29.450241 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:32:36.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:32:36.411969 3798194 memory.go:184] no items to output this cycle I1121 22:32:36.412123 3798194 cpu.go:275] no items to output this cycle E1121 22:32:46.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:32:46.413261 3798194 memory.go:184] no items to output this cycle I1121 22:32:46.413404 3798194 cpu.go:275] no items to output this cycle E1121 22:32:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:32:56.411955 3798194 memory.go:191] Add success. W1121 22:32:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:32:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:32:56.412002 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:32:56.412360 3798194 cpu.go:282] Add success. I1121 22:32:56.441817 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:32:56.442412 3798194 net.go:642] Add success. I1121 22:32:56.446392 3798194 net.go:778] primary dev: ETH0 I1121 22:32:56.446407 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:32:56.446419 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:32:57.450964 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:32:57.451146 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:32:57.451201 3798194 disk_worker.go:723] disk space is not compliant W1121 22:32:57.451204 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:32:57.459738 3798194 disk_worker.go:509] system disk:vda1 I1121 22:32:57.459784 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:32:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:32:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:32:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:32:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:32:59.471670 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:33:00.173146 3798194 disk_info.go:125] begin check local disk info of client I1121 22:33:00.176921 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:33:00.176930 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000398fc0 0xc000399000] E1121 22:33:06.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:33:06.413252 3798194 memory.go:184] no items to output this cycle I1121 22:33:06.413395 3798194 cpu.go:275] no items to output this cycle E1121 22:33:16.412874 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:33:16.412896 3798194 memory.go:184] no items to output this cycle I1121 22:33:16.413052 3798194 cpu.go:275] no items to output this cycle I1121 22:33:17.568907 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:33:17.568915 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:33:26.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:33:26.412887 3798194 memory.go:191] Add success. I1121 22:33:26.417070 3798194 cpu.go:282] Add success. I1121 22:33:26.441580 3798194 net.go:642] Add success. I1121 22:33:26.443049 3798194 net.go:778] primary dev: ETH0 I1121 22:33:26.443065 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:33:26.443077 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:33:29.449746 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:33:29.449819 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:33:29.449846 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:33:36.412623 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:33:36.412646 3798194 memory.go:184] no items to output this cycle I1121 22:33:36.412798 3798194 cpu.go:275] no items to output this cycle E1121 22:33:46.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:33:46.412261 3798194 memory.go:184] no items to output this cycle I1121 22:33:46.412409 3798194 cpu.go:275] no items to output this cycle E1121 22:33:56.412224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:33:56.412261 3798194 memory.go:191] Add success. W1121 22:33:56.412295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:33:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:33:56.412310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:33:56.416725 3798194 cpu.go:282] Add success. I1121 22:33:56.441133 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:33:56.441616 3798194 net.go:642] Add success. I1121 22:33:56.443135 3798194 net.go:778] primary dev: ETH0 I1121 22:33:56.443150 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:33:56.443316 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:33:57.443682 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:33:57.450556 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:33:57.450613 3798194 disk_worker.go:723] disk space is not compliant W1121 22:33:57.450616 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:33:57.458903 3798194 disk_worker.go:509] system disk:vda1 I1121 22:33:57.458946 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:33:58.446611 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:33:59.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:33:59.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:33:59.450497 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:33:59.471035 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:34:00.179146 3798194 disk_info.go:125] begin check local disk info of client I1121 22:34:00.182999 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:34:00.183008 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004d6240 0xc0004d6280] E1121 22:34:06.412903 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:34:06.412926 3798194 memory.go:184] no items to output this cycle I1121 22:34:06.413088 3798194 cpu.go:275] no items to output this cycle E1121 22:34:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:34:16.411985 3798194 memory.go:184] no items to output this cycle I1121 22:34:16.412156 3798194 cpu.go:275] no items to output this cycle E1121 22:34:26.411934 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:34:26.411971 3798194 memory.go:191] Add success. I1121 22:34:26.412340 3798194 cpu.go:282] Add success. I1121 22:34:26.441973 3798194 net.go:642] Add success. I1121 22:34:26.445931 3798194 net.go:778] primary dev: ETH0 I1121 22:34:26.445946 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:34:26.445964 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:34:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:34:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:34:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:34:36.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:34:36.412885 3798194 memory.go:184] no items to output this cycle I1121 22:34:36.413027 3798194 cpu.go:275] no items to output this cycle E1121 22:34:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:34:46.413253 3798194 memory.go:184] no items to output this cycle I1121 22:34:46.413406 3798194 cpu.go:275] no items to output this cycle E1121 22:34:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:34:56.411948 3798194 memory.go:191] Add success. W1121 22:34:56.411975 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:34:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:34:56.411991 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:34:56.412349 3798194 cpu.go:282] Add success. I1121 22:34:56.441804 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:34:56.442408 3798194 net.go:642] Add success. I1121 22:34:56.446306 3798194 net.go:778] primary dev: ETH0 I1121 22:34:56.446320 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:34:56.446332 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:34:57.450565 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:34:57.450720 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:34:57.450776 3798194 disk_worker.go:723] disk space is not compliant W1121 22:34:57.450780 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:34:57.459123 3798194 disk_worker.go:509] system disk:vda1 I1121 22:34:57.459165 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:34:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:34:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:34:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:34:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:34:59.471673 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:35:00.183106 3798194 disk_info.go:125] begin check local disk info of client I1121 22:35:00.186887 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:35:00.186895 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036d440 0xc00036d480] E1121 22:35:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:35:06.412253 3798194 memory.go:184] no items to output this cycle I1121 22:35:06.412394 3798194 cpu.go:275] no items to output this cycle E1121 22:35:16.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:35:16.413266 3798194 memory.go:184] no items to output this cycle I1121 22:35:16.413413 3798194 cpu.go:275] no items to output this cycle E1121 22:35:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:35:26.411959 3798194 memory.go:191] Add success. I1121 22:35:26.412351 3798194 cpu.go:282] Add success. I1121 22:35:26.441885 3798194 net.go:642] Add success. I1121 22:35:26.445899 3798194 net.go:778] primary dev: ETH0 I1121 22:35:26.445915 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:35:26.445928 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:35:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:35:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:35:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:35:36.413170 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:35:36.413191 3798194 memory.go:184] no items to output this cycle I1121 22:35:36.413337 3798194 cpu.go:275] no items to output this cycle E1121 22:35:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:35:46.411977 3798194 memory.go:184] no items to output this cycle I1121 22:35:46.412131 3798194 cpu.go:275] no items to output this cycle E1121 22:35:56.411908 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:35:56.411941 3798194 memory.go:191] Add success. W1121 22:35:56.411967 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:35:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:35:56.411982 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:35:56.412330 3798194 cpu.go:282] Add success. I1121 22:35:56.441566 3798194 event_worker.go:152] Polling the log file for events... I1121 22:35:56.444844 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:35:56.445329 3798194 net.go:642] Add success. I1121 22:35:56.449371 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795269956, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 22:35:56.449481 3798194 net.go:778] primary dev: ETH0 I1121 22:35:56.449495 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:35:56.449506 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:35:56.458987 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 22:35:57.445488 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 22:35:57.445503 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:35:57.445706 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:35:57.445766 3798194 disk_worker.go:723] disk space is not compliant W1121 22:35:57.445769 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:35:57.447838 3798194 disk_worker.go:509] system disk:vda1 I1121 22:35:57.447879 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 22:35:58.448480 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 22:35:58.448494 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:35:59.449866 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:35:59.449918 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:35:59.449938 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:35:59.450059 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 22:35:59.471504 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:36:00.187001 3798194 disk_info.go:125] begin check local disk info of client I1121 22:36:00.190831 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:36:00.190840 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e9040 0xc0002e9080] E1121 22:36:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:36:06.413256 3798194 memory.go:184] no items to output this cycle I1121 22:36:06.413401 3798194 cpu.go:275] no items to output this cycle E1121 22:36:16.413222 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:36:16.413245 3798194 memory.go:184] no items to output this cycle I1121 22:36:16.413391 3798194 cpu.go:275] no items to output this cycle I1121 22:36:17.569970 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:36:17.569980 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:36:26.413554 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:36:26.413590 3798194 memory.go:191] Add success. I1121 22:36:26.418100 3798194 cpu.go:282] Add success. I1121 22:36:26.441736 3798194 net.go:642] Add success. I1121 22:36:26.443339 3798194 net.go:778] primary dev: ETH0 I1121 22:36:26.443355 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:36:26.443369 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:36:29.452145 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:36:29.452245 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:36:29.452278 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:36:36.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:36:36.411969 3798194 memory.go:184] no items to output this cycle I1121 22:36:36.412108 3798194 cpu.go:275] no items to output this cycle E1121 22:36:46.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:36:46.413268 3798194 memory.go:184] no items to output this cycle I1121 22:36:46.413432 3798194 cpu.go:275] no items to output this cycle E1121 22:36:56.412251 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:36:56.412293 3798194 memory.go:191] Add success. W1121 22:36:56.412328 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:36:56.412339 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:36:56.412342 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:36:56.417059 3798194 cpu.go:282] Add success. I1121 22:36:56.441628 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:36:56.442094 3798194 net.go:642] Add success. I1121 22:36:56.443781 3798194 net.go:778] primary dev: ETH0 I1121 22:36:56.443796 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:36:56.443809 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:36:57.454579 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:36:57.454794 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:36:57.454807 3798194 disk_worker.go:723] disk space is not compliant W1121 22:36:57.454811 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:36:57.470810 3798194 disk_worker.go:509] system disk:vda1 I1121 22:36:57.470852 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:36:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:36:59.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:36:59.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:36:59.450493 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:36:59.471017 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:37:00.193146 3798194 disk_info.go:125] begin check local disk info of client I1121 22:37:00.196990 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:37:00.196999 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0980 0xc0002e09c0] E1121 22:37:06.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:37:06.411982 3798194 memory.go:184] no items to output this cycle I1121 22:37:06.412142 3798194 cpu.go:275] no items to output this cycle E1121 22:37:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:37:16.413263 3798194 memory.go:184] no items to output this cycle I1121 22:37:16.413407 3798194 cpu.go:275] no items to output this cycle E1121 22:37:26.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:37:26.411944 3798194 memory.go:191] Add success. I1121 22:37:26.412286 3798194 cpu.go:282] Add success. I1121 22:37:26.441679 3798194 net.go:642] Add success. I1121 22:37:26.445718 3798194 net.go:778] primary dev: ETH0 I1121 22:37:26.445732 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:37:26.445745 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:37:29.449675 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:37:29.449750 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:37:29.449774 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:37:36.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:37:36.413253 3798194 memory.go:184] no items to output this cycle I1121 22:37:36.413394 3798194 cpu.go:275] no items to output this cycle E1121 22:37:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:37:46.411982 3798194 memory.go:184] no items to output this cycle I1121 22:37:46.412142 3798194 cpu.go:275] no items to output this cycle E1121 22:37:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:37:56.412268 3798194 memory.go:191] Add success. W1121 22:37:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:37:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:37:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:37:56.416740 3798194 cpu.go:282] Add success. I1121 22:37:56.441134 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:37:56.441616 3798194 net.go:642] Add success. I1121 22:37:56.443166 3798194 net.go:778] primary dev: ETH0 I1121 22:37:56.443180 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:37:56.443199 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:37:57.451441 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:37:57.451617 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:37:57.451672 3798194 disk_worker.go:723] disk space is not compliant W1121 22:37:57.451675 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:37:57.460904 3798194 disk_worker.go:509] system disk:vda1 I1121 22:37:57.460944 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:37:58.446371 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:37:59.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:37:59.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:37:59.450602 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:37:59.470727 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:38:00.200151 3798194 disk_info.go:125] begin check local disk info of client I1121 22:38:00.203949 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:38:00.203957 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000492140 0xc000492180] E1121 22:38:06.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:38:06.411980 3798194 memory.go:184] no items to output this cycle I1121 22:38:06.412145 3798194 cpu.go:275] no items to output this cycle E1121 22:38:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:38:16.411977 3798194 memory.go:184] no items to output this cycle I1121 22:38:16.412138 3798194 cpu.go:275] no items to output this cycle E1121 22:38:26.412847 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:38:26.412883 3798194 memory.go:191] Add success. I1121 22:38:26.417316 3798194 cpu.go:282] Add success. I1121 22:38:26.441731 3798194 net.go:642] Add success. I1121 22:38:26.443233 3798194 net.go:778] primary dev: ETH0 I1121 22:38:26.443248 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:38:26.443261 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:38:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:38:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:38:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:38:36.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:38:36.412264 3798194 memory.go:184] no items to output this cycle I1121 22:38:36.412402 3798194 cpu.go:275] no items to output this cycle E1121 22:38:46.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:38:46.412880 3798194 memory.go:184] no items to output this cycle I1121 22:38:46.413020 3798194 cpu.go:275] no items to output this cycle E1121 22:38:56.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:38:56.412263 3798194 memory.go:191] Add success. W1121 22:38:56.412291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:38:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:38:56.412305 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:38:56.416880 3798194 cpu.go:282] Add success. I1121 22:38:56.441272 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:38:56.441739 3798194 net.go:642] Add success. I1121 22:38:56.443273 3798194 net.go:778] primary dev: ETH0 I1121 22:38:56.443288 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:38:56.443302 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:38:57.450867 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:38:57.451033 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:38:57.451087 3798194 disk_worker.go:723] disk space is not compliant W1121 22:38:57.451090 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:38:57.460005 3798194 disk_worker.go:509] system disk:vda1 I1121 22:38:57.460050 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:38:58.445698 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:38:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:38:59.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:38:59.450506 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:38:59.471052 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:39:00.206138 3798194 disk_info.go:125] begin check local disk info of client I1121 22:39:00.209997 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:39:00.210008 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000399500 0xc000399540] E1121 22:39:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:39:06.411981 3798194 memory.go:184] no items to output this cycle I1121 22:39:06.412148 3798194 cpu.go:275] no items to output this cycle E1121 22:39:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:39:16.411978 3798194 memory.go:184] no items to output this cycle I1121 22:39:16.412136 3798194 cpu.go:275] no items to output this cycle I1121 22:39:17.571069 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:39:17.571077 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:39:26.411633 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:39:26.411668 3798194 memory.go:191] Add success. I1121 22:39:26.411991 3798194 cpu.go:282] Add success. I1121 22:39:26.441606 3798194 net.go:642] Add success. I1121 22:39:26.445704 3798194 net.go:778] primary dev: ETH0 I1121 22:39:26.445721 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:39:26.445734 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:39:29.450560 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:39:29.450640 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:39:29.450664 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:39:36.412325 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:39:36.412348 3798194 memory.go:184] no items to output this cycle I1121 22:39:36.412703 3798194 cpu.go:275] no items to output this cycle E1121 22:39:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:39:46.413254 3798194 memory.go:184] no items to output this cycle I1121 22:39:46.413403 3798194 cpu.go:275] no items to output this cycle E1121 22:39:56.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:39:56.412263 3798194 memory.go:191] Add success. W1121 22:39:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:39:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:39:56.412312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:39:56.416687 3798194 cpu.go:282] Add success. I1121 22:39:56.441074 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:39:56.441577 3798194 net.go:642] Add success. I1121 22:39:56.443240 3798194 net.go:778] primary dev: ETH0 I1121 22:39:56.443254 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:39:56.443268 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:39:57.450786 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:39:57.451038 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:39:57.451055 3798194 disk_worker.go:723] disk space is not compliant W1121 22:39:57.451058 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:39:57.459800 3798194 disk_worker.go:509] system disk:vda1 I1121 22:39:57.459843 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:39:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:39:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:39:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:39:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:39:59.471676 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:40:00.210118 3798194 disk_info.go:125] begin check local disk info of client I1121 22:40:00.213949 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:40:00.213958 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000398b80 0xc000398bc0] E1121 22:40:06.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:40:06.412886 3798194 memory.go:184] no items to output this cycle I1121 22:40:06.413030 3798194 cpu.go:275] no items to output this cycle E1121 22:40:16.413552 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:40:16.413576 3798194 memory.go:184] no items to output this cycle I1121 22:40:16.413738 3798194 cpu.go:275] no items to output this cycle E1121 22:40:26.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:40:26.411957 3798194 memory.go:191] Add success. I1121 22:40:26.412314 3798194 cpu.go:282] Add success. I1121 22:40:26.441931 3798194 net.go:642] Add success. I1121 22:40:26.445864 3798194 net.go:778] primary dev: ETH0 I1121 22:40:26.445879 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:40:26.445893 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:40:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:40:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:40:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:40:36.412092 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:40:36.412134 3798194 memory.go:184] no items to output this cycle I1121 22:40:36.412280 3798194 cpu.go:275] no items to output this cycle E1121 22:40:46.412865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:40:46.412890 3798194 memory.go:184] no items to output this cycle I1121 22:40:46.413042 3798194 cpu.go:275] no items to output this cycle E1121 22:40:56.412107 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:40:56.412156 3798194 memory.go:191] Add success. W1121 22:40:56.412192 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:40:56.412203 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:40:56.412206 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:40:56.416677 3798194 cpu.go:282] Add success. I1121 22:40:56.444232 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:40:56.444707 3798194 net.go:642] Add success. I1121 22:40:56.446196 3798194 net.go:778] primary dev: ETH0 I1121 22:40:56.446212 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:40:56.446225 3798194 net.go:659] Add eni metrics success, count: 7 E1121 22:40:57.450521 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 22:40:57.450535 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:40:57.450732 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:40:57.450741 3798194 disk_worker.go:723] disk space is not compliant W1121 22:40:57.450744 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:40:57.459935 3798194 disk_worker.go:509] system disk:vda1 I1121 22:40:57.459976 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 22:40:58.448043 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 22:40:58.448058 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 22:40:59.448613 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 22:40:59.449687 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:40:59.449742 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:40:59.449761 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:40:59.471181 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:41:00.214059 3798194 disk_info.go:125] begin check local disk info of client I1121 22:41:00.217972 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:41:00.217981 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a7a00 0xc0004a7a40] E1121 22:41:06.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:41:06.411987 3798194 memory.go:184] no items to output this cycle I1121 22:41:06.412145 3798194 cpu.go:275] no items to output this cycle E1121 22:41:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:41:16.413258 3798194 memory.go:184] no items to output this cycle I1121 22:41:16.413406 3798194 cpu.go:275] no items to output this cycle E1121 22:41:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:41:26.411959 3798194 memory.go:191] Add success. I1121 22:41:26.412308 3798194 cpu.go:282] Add success. I1121 22:41:26.442053 3798194 net.go:642] Add success. I1121 22:41:26.445859 3798194 net.go:778] primary dev: ETH0 I1121 22:41:26.445876 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:41:26.445890 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:41:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:41:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:41:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:41:36.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:41:36.412267 3798194 memory.go:184] no items to output this cycle I1121 22:41:36.412406 3798194 cpu.go:275] no items to output this cycle E1121 22:41:46.412246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:41:46.412271 3798194 memory.go:184] no items to output this cycle I1121 22:41:46.412420 3798194 cpu.go:275] no items to output this cycle E1121 22:41:56.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:41:56.412268 3798194 memory.go:191] Add success. W1121 22:41:56.412301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:41:56.412321 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:41:56.412324 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:41:56.416672 3798194 cpu.go:282] Add success. I1121 22:41:56.441242 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:41:56.441807 3798194 net.go:642] Add success. I1121 22:41:56.443277 3798194 net.go:778] primary dev: ETH0 I1121 22:41:56.443292 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:41:56.443313 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:41:57.454850 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:41:57.455016 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:41:57.455070 3798194 disk_worker.go:723] disk space is not compliant W1121 22:41:57.455073 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:41:57.465357 3798194 disk_worker.go:509] system disk:vda1 I1121 22:41:57.465410 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:41:58.446435 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:41:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:41:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:41:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:41:59.471693 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:42:00.221138 3798194 disk_info.go:125] begin check local disk info of client I1121 22:42:00.224994 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:42:00.225003 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fac80 0xc0003facc0] E1121 22:42:06.411239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:42:06.411261 3798194 memory.go:184] no items to output this cycle I1121 22:42:06.411401 3798194 cpu.go:275] no items to output this cycle E1121 22:42:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:42:16.411980 3798194 memory.go:184] no items to output this cycle I1121 22:42:16.412144 3798194 cpu.go:275] no items to output this cycle I1121 22:42:17.571240 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:42:17.571248 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:42:26.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:42:26.412887 3798194 memory.go:191] Add success. I1121 22:42:26.417102 3798194 cpu.go:282] Add success. I1121 22:42:26.441307 3798194 net.go:642] Add success. I1121 22:42:26.442783 3798194 net.go:778] primary dev: ETH0 I1121 22:42:26.442798 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:42:26.442813 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:42:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:42:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:42:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:42:36.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:42:36.413253 3798194 memory.go:184] no items to output this cycle I1121 22:42:36.413397 3798194 cpu.go:275] no items to output this cycle E1121 22:42:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:42:46.411970 3798194 memory.go:184] no items to output this cycle I1121 22:42:46.412121 3798194 cpu.go:275] no items to output this cycle E1121 22:42:56.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:42:56.413262 3798194 memory.go:191] Add success. W1121 22:42:56.413290 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:42:56.413301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:42:56.413304 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:42:56.417585 3798194 cpu.go:282] Add success. I1121 22:42:56.440918 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:42:56.441464 3798194 net.go:642] Add success. I1121 22:42:56.442935 3798194 net.go:778] primary dev: ETH0 I1121 22:42:56.442949 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:42:56.442961 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:42:57.449151 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:42:57.449345 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:42:57.449400 3798194 disk_worker.go:723] disk space is not compliant W1121 22:42:57.449403 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:42:57.452909 3798194 disk_worker.go:509] system disk:vda1 I1121 22:42:57.452952 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:42:58.445987 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:42:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:42:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:42:59.450148 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:42:59.471643 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:43:00.228402 3798194 disk_info.go:125] begin check local disk info of client I1121 22:43:00.232280 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:43:00.232295 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002f4840 0xc0002f4880] E1121 22:43:06.412571 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:43:06.412593 3798194 memory.go:184] no items to output this cycle I1121 22:43:06.412735 3798194 cpu.go:275] no items to output this cycle E1121 22:43:16.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:43:16.411987 3798194 memory.go:184] no items to output this cycle I1121 22:43:16.412150 3798194 cpu.go:275] no items to output this cycle E1121 22:43:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:43:26.411948 3798194 memory.go:191] Add success. I1121 22:43:26.412294 3798194 cpu.go:282] Add success. I1121 22:43:26.441752 3798194 net.go:642] Add success. I1121 22:43:26.445544 3798194 net.go:778] primary dev: ETH0 I1121 22:43:26.445559 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:43:26.445575 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:43:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:43:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:43:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:43:36.413160 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:43:36.413186 3798194 memory.go:184] no items to output this cycle I1121 22:43:36.413340 3798194 cpu.go:275] no items to output this cycle E1121 22:43:46.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:43:46.413256 3798194 memory.go:184] no items to output this cycle I1121 22:43:46.413401 3798194 cpu.go:275] no items to output this cycle E1121 22:43:56.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:43:56.413268 3798194 memory.go:191] Add success. W1121 22:43:56.413300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:43:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:43:56.413315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:43:56.417626 3798194 cpu.go:282] Add success. I1121 22:43:56.441685 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:43:56.442838 3798194 net.go:642] Add success. I1121 22:43:56.444481 3798194 net.go:778] primary dev: ETH0 I1121 22:43:56.444495 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:43:56.444507 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:43:57.446283 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:43:57.446499 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:43:57.446575 3798194 disk_worker.go:723] disk space is not compliant W1121 22:43:57.446578 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:43:57.455348 3798194 disk_worker.go:509] system disk:vda1 I1121 22:43:57.455397 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:43:58.446383 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:43:59.450482 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:43:59.450563 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:43:59.450589 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:43:59.471102 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:44:00.235969 3798194 disk_info.go:125] begin check local disk info of client I1121 22:44:00.239792 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:44:00.239801 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004ab440 0xc0004ab480] E1121 22:44:06.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:44:06.411993 3798194 memory.go:184] no items to output this cycle I1121 22:44:06.412153 3798194 cpu.go:275] no items to output this cycle E1121 22:44:16.412249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:44:16.412270 3798194 memory.go:184] no items to output this cycle I1121 22:44:16.412441 3798194 cpu.go:275] no items to output this cycle E1121 22:44:26.412842 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:44:26.412880 3798194 memory.go:191] Add success. I1121 22:44:26.417436 3798194 cpu.go:282] Add success. I1121 22:44:26.442261 3798194 net.go:642] Add success. I1121 22:44:26.443711 3798194 net.go:778] primary dev: ETH0 I1121 22:44:26.443725 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:44:26.443738 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:44:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:44:29.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:44:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:44:36.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:44:36.413252 3798194 memory.go:184] no items to output this cycle I1121 22:44:36.413398 3798194 cpu.go:275] no items to output this cycle E1121 22:44:46.412289 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:44:46.412320 3798194 memory.go:184] no items to output this cycle I1121 22:44:46.412523 3798194 cpu.go:275] no items to output this cycle E1121 22:44:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:44:56.411959 3798194 memory.go:191] Add success. W1121 22:44:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:44:56.412005 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:44:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:44:56.412359 3798194 cpu.go:282] Add success. I1121 22:44:56.441816 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:44:56.442334 3798194 net.go:642] Add success. I1121 22:44:56.446317 3798194 net.go:778] primary dev: ETH0 I1121 22:44:56.446333 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:44:56.446345 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:44:57.450637 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:44:57.450858 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:44:57.450869 3798194 disk_worker.go:723] disk space is not compliant W1121 22:44:57.450872 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:44:57.459166 3798194 disk_worker.go:509] system disk:vda1 I1121 22:44:57.459213 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:44:58.446137 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:44:59.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:44:59.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:44:59.450152 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:44:59.471685 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:45:00.243151 3798194 disk_info.go:125] begin check local disk info of client I1121 22:45:00.246980 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:45:00.246989 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037e1c0 0xc00037e200] E1121 22:45:06.412094 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:45:06.412133 3798194 memory.go:184] no items to output this cycle I1121 22:45:06.412281 3798194 cpu.go:275] no items to output this cycle E1121 22:45:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:45:16.411981 3798194 memory.go:184] no items to output this cycle I1121 22:45:16.412136 3798194 cpu.go:275] no items to output this cycle I1121 22:45:17.571409 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:45:17.571418 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:45:26.413251 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:45:26.413287 3798194 memory.go:191] Add success. I1121 22:45:26.417774 3798194 cpu.go:282] Add success. I1121 22:45:26.441562 3798194 net.go:642] Add success. I1121 22:45:26.442980 3798194 net.go:778] primary dev: ETH0 I1121 22:45:26.442994 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:45:26.443008 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:45:29.450512 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:45:29.450598 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:45:29.450625 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:45:36.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:45:36.411991 3798194 memory.go:184] no items to output this cycle I1121 22:45:36.412149 3798194 cpu.go:275] no items to output this cycle E1121 22:45:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:45:46.413255 3798194 memory.go:184] no items to output this cycle I1121 22:45:46.413402 3798194 cpu.go:275] no items to output this cycle E1121 22:45:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:45:56.411945 3798194 memory.go:191] Add success. W1121 22:45:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:45:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:45:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:45:56.412360 3798194 cpu.go:282] Add success. I1121 22:45:56.443813 3798194 event_worker.go:152] Polling the log file for events... I1121 22:45:56.444025 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:45:56.444526 3798194 net.go:642] Add success. I1121 22:45:56.448677 3798194 net.go:778] primary dev: ETH0 I1121 22:45:56.448691 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:45:56.448703 3798194 net.go:659] Add eni metrics success, count: 7 E1121 22:45:57.450614 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 22:45:57.450628 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:45:57.450795 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:45:57.450847 3798194 disk_worker.go:723] disk space is not compliant W1121 22:45:57.450850 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:45:57.459178 3798194 disk_worker.go:509] system disk:vda1 I1121 22:45:57.459217 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 22:45:58.448418 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 22:45:58.448433 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 22:45:59.448616 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 22:45:59.449691 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:45:59.449742 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:45:59.449761 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:45:59.471279 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:46:00.250134 3798194 disk_info.go:125] begin check local disk info of client I1121 22:46:00.253953 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:46:00.253963 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003559c0 0xc000355a00] E1121 22:46:06.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:46:06.411978 3798194 memory.go:184] no items to output this cycle I1121 22:46:06.412130 3798194 cpu.go:275] no items to output this cycle E1121 22:46:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:46:16.411983 3798194 memory.go:184] no items to output this cycle I1121 22:46:16.412142 3798194 cpu.go:275] no items to output this cycle E1121 22:46:26.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:46:26.411953 3798194 memory.go:191] Add success. I1121 22:46:26.412336 3798194 cpu.go:282] Add success. I1121 22:46:26.441338 3798194 net.go:642] Add success. I1121 22:46:26.445510 3798194 net.go:778] primary dev: ETH0 I1121 22:46:26.445526 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:46:26.445539 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:46:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:46:29.450134 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:46:29.450163 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:46:36.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:46:36.411969 3798194 memory.go:184] no items to output this cycle I1121 22:46:36.412126 3798194 cpu.go:275] no items to output this cycle E1121 22:46:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:46:46.413262 3798194 memory.go:184] no items to output this cycle I1121 22:46:46.413411 3798194 cpu.go:275] no items to output this cycle E1121 22:46:56.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:46:56.412898 3798194 memory.go:191] Add success. W1121 22:46:56.412931 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:46:56.412943 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:46:56.412946 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:46:56.417820 3798194 cpu.go:282] Add success. I1121 22:46:56.441329 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:46:56.441901 3798194 net.go:642] Add success. I1121 22:46:56.443615 3798194 net.go:778] primary dev: ETH0 I1121 22:46:56.443631 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:46:56.443643 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:46:57.450969 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:46:57.451203 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:46:57.451217 3798194 disk_worker.go:723] disk space is not compliant W1121 22:46:57.451220 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:46:57.459997 3798194 disk_worker.go:509] system disk:vda1 I1121 22:46:57.460037 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:46:58.445974 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:46:59.450082 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:46:59.450165 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:46:59.450192 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:46:59.471714 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:47:00.254066 3798194 disk_info.go:125] begin check local disk info of client I1121 22:47:00.257918 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:47:00.257928 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000237e00 0xc000237e40] E1121 22:47:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:47:06.413257 3798194 memory.go:184] no items to output this cycle I1121 22:47:06.413400 3798194 cpu.go:275] no items to output this cycle E1121 22:47:16.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:47:16.413252 3798194 memory.go:184] no items to output this cycle I1121 22:47:16.413408 3798194 cpu.go:275] no items to output this cycle E1121 22:47:26.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:47:26.413270 3798194 memory.go:191] Add success. I1121 22:47:26.417313 3798194 cpu.go:282] Add success. I1121 22:47:26.441746 3798194 net.go:642] Add success. I1121 22:47:26.443088 3798194 net.go:778] primary dev: ETH0 I1121 22:47:26.443101 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:47:26.443127 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:47:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:47:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:47:29.450518 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:47:36.413663 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:47:36.413686 3798194 memory.go:184] no items to output this cycle I1121 22:47:36.413844 3798194 cpu.go:275] no items to output this cycle E1121 22:47:46.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:47:46.413250 3798194 memory.go:184] no items to output this cycle I1121 22:47:46.413396 3798194 cpu.go:275] no items to output this cycle E1121 22:47:56.412252 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:47:56.412291 3798194 memory.go:191] Add success. W1121 22:47:56.412327 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:47:56.412338 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:47:56.412341 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:47:56.416870 3798194 cpu.go:282] Add success. I1121 22:47:56.441290 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:47:56.441769 3798194 net.go:642] Add success. I1121 22:47:56.443496 3798194 net.go:778] primary dev: ETH0 I1121 22:47:56.443510 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:47:56.443525 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:47:57.451015 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:47:57.451244 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:47:57.451255 3798194 disk_worker.go:723] disk space is not compliant W1121 22:47:57.451258 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:47:57.460021 3798194 disk_worker.go:509] system disk:vda1 I1121 22:47:57.460062 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:47:58.446459 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:47:59.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:47:59.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:47:59.450501 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:47:59.471030 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:48:00.261014 3798194 disk_info.go:125] begin check local disk info of client I1121 22:48:00.264871 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:48:00.264879 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030f740 0xc00030f780] E1121 22:48:06.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:48:06.412878 3798194 memory.go:184] no items to output this cycle I1121 22:48:06.413027 3798194 cpu.go:275] no items to output this cycle E1121 22:48:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:48:16.411975 3798194 memory.go:184] no items to output this cycle I1121 22:48:16.412131 3798194 cpu.go:275] no items to output this cycle I1121 22:48:17.571573 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:48:17.571581 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:48:26.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:48:26.412883 3798194 memory.go:191] Add success. I1121 22:48:26.417503 3798194 cpu.go:282] Add success. I1121 22:48:26.442201 3798194 net.go:642] Add success. I1121 22:48:26.443645 3798194 net.go:778] primary dev: ETH0 I1121 22:48:26.443661 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:48:26.443673 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:48:29.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:48:29.450131 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:48:29.450164 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:48:36.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:48:36.413257 3798194 memory.go:184] no items to output this cycle I1121 22:48:36.413401 3798194 cpu.go:275] no items to output this cycle E1121 22:48:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:48:46.411981 3798194 memory.go:184] no items to output this cycle I1121 22:48:46.412138 3798194 cpu.go:275] no items to output this cycle E1121 22:48:56.412869 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:48:56.412904 3798194 memory.go:191] Add success. W1121 22:48:56.412940 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:48:56.412953 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:48:56.412956 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:48:56.416949 3798194 cpu.go:282] Add success. I1121 22:48:56.441332 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:48:56.441804 3798194 net.go:642] Add success. I1121 22:48:56.443367 3798194 net.go:778] primary dev: ETH0 I1121 22:48:56.443389 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:48:56.443404 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:48:57.450760 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:48:57.450969 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:48:57.450980 3798194 disk_worker.go:723] disk space is not compliant W1121 22:48:57.450983 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:48:57.460501 3798194 disk_worker.go:509] system disk:vda1 I1121 22:48:57.460547 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:48:58.446375 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:48:59.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:48:59.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:48:59.450512 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:48:59.471033 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:49:00.267868 3798194 disk_info.go:125] begin check local disk info of client I1121 22:49:00.271772 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:49:00.271780 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00045f880 0xc00045f8c0] E1121 22:49:06.413530 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:49:06.413552 3798194 memory.go:184] no items to output this cycle I1121 22:49:06.413694 3798194 cpu.go:275] no items to output this cycle E1121 22:49:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:49:16.411984 3798194 memory.go:184] no items to output this cycle I1121 22:49:16.412144 3798194 cpu.go:275] no items to output this cycle E1121 22:49:26.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:49:26.413277 3798194 memory.go:191] Add success. I1121 22:49:26.417408 3798194 cpu.go:282] Add success. I1121 22:49:26.441934 3798194 net.go:642] Add success. I1121 22:49:26.443649 3798194 net.go:778] primary dev: ETH0 I1121 22:49:26.443663 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:49:26.443676 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:49:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:49:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:49:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:49:36.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:49:36.413252 3798194 memory.go:184] no items to output this cycle I1121 22:49:36.413395 3798194 cpu.go:275] no items to output this cycle E1121 22:49:46.413311 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:49:46.413337 3798194 memory.go:184] no items to output this cycle I1121 22:49:46.413488 3798194 cpu.go:275] no items to output this cycle E1121 22:49:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:49:56.411951 3798194 memory.go:191] Add success. W1121 22:49:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:49:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:49:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:49:56.412342 3798194 cpu.go:282] Add success. I1121 22:49:56.441817 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:49:56.442318 3798194 net.go:642] Add success. I1121 22:49:56.446824 3798194 net.go:778] primary dev: ETH0 I1121 22:49:56.446839 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:49:56.446851 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:49:57.450738 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:49:57.450952 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:49:57.450963 3798194 disk_worker.go:723] disk space is not compliant W1121 22:49:57.450966 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:49:57.459966 3798194 disk_worker.go:509] system disk:vda1 I1121 22:49:57.460009 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:49:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:49:59.450105 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:49:59.450189 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:49:59.450223 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:49:59.470695 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:50:00.275134 3798194 disk_info.go:125] begin check local disk info of client I1121 22:50:00.278959 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:50:00.278968 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003bc300 0xc0003bc340] E1121 22:50:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:50:06.413255 3798194 memory.go:184] no items to output this cycle I1121 22:50:06.413398 3798194 cpu.go:275] no items to output this cycle E1121 22:50:16.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:50:16.412878 3798194 memory.go:184] no items to output this cycle I1121 22:50:16.413027 3798194 cpu.go:275] no items to output this cycle E1121 22:50:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:50:26.411959 3798194 memory.go:191] Add success. I1121 22:50:26.412329 3798194 cpu.go:282] Add success. I1121 22:50:26.441958 3798194 net.go:642] Add success. I1121 22:50:26.446631 3798194 net.go:778] primary dev: ETH0 I1121 22:50:26.446647 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:50:26.446660 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:50:29.449889 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:50:29.449965 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:50:29.449993 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:50:36.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:50:36.413258 3798194 memory.go:184] no items to output this cycle I1121 22:50:36.413400 3798194 cpu.go:275] no items to output this cycle E1121 22:50:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:50:46.411977 3798194 memory.go:184] no items to output this cycle I1121 22:50:46.412143 3798194 cpu.go:275] no items to output this cycle E1121 22:50:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:50:56.411956 3798194 memory.go:191] Add success. W1121 22:50:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:50:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:50:56.412002 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:50:56.412355 3798194 cpu.go:282] Add success. I1121 22:50:56.445052 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:50:56.445569 3798194 net.go:642] Add success. I1121 22:50:56.449657 3798194 net.go:778] primary dev: ETH0 I1121 22:50:56.449673 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:50:56.449685 3798194 net.go:659] Add eni metrics success, count: 7 E1121 22:50:57.450692 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 22:50:57.450707 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:50:57.450876 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:50:57.450931 3798194 disk_worker.go:723] disk space is not compliant W1121 22:50:57.450934 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:50:57.461914 3798194 disk_worker.go:509] system disk:vda1 I1121 22:50:57.461957 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 22:50:58.448199 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 22:50:58.448213 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 22:50:59.448600 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 22:50:59.449674 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:50:59.449723 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:50:59.449744 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:50:59.471245 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:51:00.279071 3798194 disk_info.go:125] begin check local disk info of client I1121 22:51:00.282895 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:51:00.282903 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000311540 0xc000311580] E1121 22:51:06.412758 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:51:06.412786 3798194 memory.go:184] no items to output this cycle I1121 22:51:06.412937 3798194 cpu.go:275] no items to output this cycle E1121 22:51:16.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:51:16.413247 3798194 memory.go:184] no items to output this cycle I1121 22:51:16.413391 3798194 cpu.go:275] no items to output this cycle I1121 22:51:17.571739 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:51:17.571748 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:51:26.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:51:26.412887 3798194 memory.go:191] Add success. I1121 22:51:26.416958 3798194 cpu.go:282] Add success. I1121 22:51:26.441487 3798194 net.go:642] Add success. I1121 22:51:26.443226 3798194 net.go:778] primary dev: ETH0 I1121 22:51:26.443241 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:51:26.443254 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:51:29.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:51:29.450467 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:51:29.450495 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:51:36.412810 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:51:36.412833 3798194 memory.go:184] no items to output this cycle I1121 22:51:36.412977 3798194 cpu.go:275] no items to output this cycle E1121 22:51:46.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:51:46.413352 3798194 memory.go:184] no items to output this cycle I1121 22:51:46.413683 3798194 cpu.go:275] no items to output this cycle E1121 22:51:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:51:56.411953 3798194 memory.go:191] Add success. W1121 22:51:56.411981 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:51:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:51:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:51:56.412364 3798194 cpu.go:282] Add success. I1121 22:51:56.441829 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:51:56.442338 3798194 net.go:642] Add success. I1121 22:51:56.446836 3798194 net.go:778] primary dev: ETH0 I1121 22:51:56.446850 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:51:56.446863 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:51:57.450794 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:51:57.450989 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:51:57.451000 3798194 disk_worker.go:723] disk space is not compliant W1121 22:51:57.451004 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:51:57.459830 3798194 disk_worker.go:509] system disk:vda1 I1121 22:51:57.459872 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:51:58.446387 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:51:59.450484 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:51:59.450563 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:51:59.450588 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:51:59.471094 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:52:00.286206 3798194 disk_info.go:125] begin check local disk info of client I1121 22:52:00.290043 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:52:00.290053 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002feec0 0xc0002fef00] E1121 22:52:06.412396 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:52:06.412419 3798194 memory.go:184] no items to output this cycle I1121 22:52:06.412566 3798194 cpu.go:275] no items to output this cycle E1121 22:52:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:52:16.413256 3798194 memory.go:184] no items to output this cycle I1121 22:52:16.413400 3798194 cpu.go:275] no items to output this cycle E1121 22:52:26.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:52:26.412267 3798194 memory.go:191] Add success. I1121 22:52:26.416809 3798194 cpu.go:282] Add success. I1121 22:52:26.441527 3798194 net.go:642] Add success. I1121 22:52:26.443074 3798194 net.go:778] primary dev: ETH0 I1121 22:52:26.443089 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:52:26.443102 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:52:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:52:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:52:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:52:36.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:52:36.411982 3798194 memory.go:184] no items to output this cycle I1121 22:52:36.412129 3798194 cpu.go:275] no items to output this cycle E1121 22:52:46.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:52:46.411988 3798194 memory.go:184] no items to output this cycle I1121 22:52:46.412154 3798194 cpu.go:275] no items to output this cycle E1121 22:52:56.412894 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:52:56.412934 3798194 memory.go:191] Add success. W1121 22:52:56.412970 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:52:56.412982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:52:56.412985 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:52:56.417690 3798194 cpu.go:282] Add success. I1121 22:52:56.441242 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:52:56.441751 3798194 net.go:642] Add success. I1121 22:52:56.443327 3798194 net.go:778] primary dev: ETH0 I1121 22:52:56.443341 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:52:56.443354 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:52:57.451142 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:52:57.451311 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:52:57.451367 3798194 disk_worker.go:723] disk space is not compliant W1121 22:52:57.451370 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:52:57.461386 3798194 disk_worker.go:509] system disk:vda1 I1121 22:52:57.461428 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:52:58.446009 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:52:59.450115 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:52:59.450208 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:52:59.450235 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:52:59.470850 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:53:00.290158 3798194 disk_info.go:125] begin check local disk info of client I1121 22:53:00.294029 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:53:00.294038 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046d6c0 0xc00046d700] E1121 22:53:06.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:53:06.412883 3798194 memory.go:184] no items to output this cycle I1121 22:53:06.413045 3798194 cpu.go:275] no items to output this cycle E1121 22:53:16.411996 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:53:16.412018 3798194 memory.go:184] no items to output this cycle I1121 22:53:16.412186 3798194 cpu.go:275] no items to output this cycle E1121 22:53:26.412957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:53:26.412992 3798194 memory.go:191] Add success. I1121 22:53:26.417318 3798194 cpu.go:282] Add success. I1121 22:53:26.441875 3798194 net.go:642] Add success. I1121 22:53:26.443332 3798194 net.go:778] primary dev: ETH0 I1121 22:53:26.443346 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:53:26.443360 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:53:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:53:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:53:29.450498 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:53:36.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:53:36.413258 3798194 memory.go:184] no items to output this cycle I1121 22:53:36.413411 3798194 cpu.go:275] no items to output this cycle E1121 22:53:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:53:46.412254 3798194 memory.go:184] no items to output this cycle I1121 22:53:46.412405 3798194 cpu.go:275] no items to output this cycle E1121 22:53:56.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:53:56.412898 3798194 memory.go:191] Add success. W1121 22:53:56.412929 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:53:56.412942 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:53:56.412945 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:53:56.417370 3798194 cpu.go:282] Add success. I1121 22:53:56.441827 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:53:56.442343 3798194 net.go:642] Add success. I1121 22:53:56.443964 3798194 net.go:778] primary dev: ETH0 I1121 22:53:56.443978 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:53:56.443991 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:53:57.450903 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:53:57.451091 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:53:57.451164 3798194 disk_worker.go:723] disk space is not compliant W1121 22:53:57.451168 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:53:57.460030 3798194 disk_worker.go:509] system disk:vda1 I1121 22:53:57.460086 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:53:58.446009 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:53:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:53:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:53:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:53:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:54:00.294142 3798194 disk_info.go:125] begin check local disk info of client I1121 22:54:00.297970 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:54:00.297979 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040a400 0xc00040a440] E1121 22:54:06.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:54:06.411975 3798194 memory.go:184] no items to output this cycle I1121 22:54:06.412125 3798194 cpu.go:275] no items to output this cycle E1121 22:54:16.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:54:16.412877 3798194 memory.go:184] no items to output this cycle I1121 22:54:16.413030 3798194 cpu.go:275] no items to output this cycle I1121 22:54:17.572877 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:54:17.572887 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:54:26.411218 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:54:26.411251 3798194 memory.go:191] Add success. I1121 22:54:26.411771 3798194 cpu.go:282] Add success. I1121 22:54:26.441316 3798194 net.go:642] Add success. I1121 22:54:26.445274 3798194 net.go:778] primary dev: ETH0 I1121 22:54:26.445291 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:54:26.445305 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:54:29.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:54:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:54:29.450148 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:54:36.412772 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:54:36.412793 3798194 memory.go:184] no items to output this cycle I1121 22:54:36.412938 3798194 cpu.go:275] no items to output this cycle E1121 22:54:46.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:54:46.413270 3798194 memory.go:184] no items to output this cycle I1121 22:54:46.413416 3798194 cpu.go:275] no items to output this cycle E1121 22:54:56.412995 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:54:56.413029 3798194 memory.go:191] Add success. W1121 22:54:56.413058 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:54:56.413069 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:54:56.413072 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:54:56.417529 3798194 cpu.go:282] Add success. I1121 22:54:56.441077 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:54:56.441566 3798194 net.go:642] Add success. I1121 22:54:56.443100 3798194 net.go:778] primary dev: ETH0 I1121 22:54:56.443124 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:54:56.443136 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:54:57.451005 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:54:57.451224 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:54:57.451236 3798194 disk_worker.go:723] disk space is not compliant W1121 22:54:57.451239 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:54:57.459975 3798194 disk_worker.go:509] system disk:vda1 I1121 22:54:57.460015 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:54:58.446371 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:54:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:54:59.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:54:59.450502 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:54:59.471017 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:55:00.298083 3798194 disk_info.go:125] begin check local disk info of client I1121 22:55:00.301911 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:55:00.301919 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000468880 0xc0004688c0] E1121 22:55:06.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:55:06.413258 3798194 memory.go:184] no items to output this cycle I1121 22:55:06.413398 3798194 cpu.go:275] no items to output this cycle E1121 22:55:16.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:55:16.413253 3798194 memory.go:184] no items to output this cycle I1121 22:55:16.413410 3798194 cpu.go:275] no items to output this cycle E1121 22:55:26.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:55:26.412271 3798194 memory.go:191] Add success. I1121 22:55:26.416467 3798194 cpu.go:282] Add success. I1121 22:55:26.442185 3798194 net.go:642] Add success. I1121 22:55:26.443637 3798194 net.go:778] primary dev: ETH0 I1121 22:55:26.443650 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:55:26.443663 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:55:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:55:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:55:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:55:36.412249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:55:36.412272 3798194 memory.go:184] no items to output this cycle I1121 22:55:36.412422 3798194 cpu.go:275] no items to output this cycle E1121 22:55:46.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:55:46.413269 3798194 memory.go:184] no items to output this cycle I1121 22:55:46.413418 3798194 cpu.go:275] no items to output this cycle E1121 22:55:56.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:55:56.413261 3798194 memory.go:191] Add success. W1121 22:55:56.413287 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:55:56.413298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:55:56.413302 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:55:56.417850 3798194 cpu.go:282] Add success. I1121 22:55:56.442468 3798194 event_worker.go:152] Polling the log file for events... I1121 22:55:56.442695 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:55:56.443194 3798194 net.go:642] Add success. I1121 22:55:56.445187 3798194 net.go:778] primary dev: ETH0 I1121 22:55:56.445202 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:55:56.445215 3798194 net.go:659] Add eni metrics success, count: 7 E1121 22:55:57.446503 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 22:55:57.446517 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:55:57.446767 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:55:57.446778 3798194 disk_worker.go:723] disk space is not compliant W1121 22:55:57.446781 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:55:57.449594 3798194 disk_worker.go:509] system disk:vda1 I1121 22:55:57.449632 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 22:55:58.448528 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 22:55:58.448543 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 22:55:59.449226 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 22:55:59.450301 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:55:59.450349 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:55:59.450368 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:55:59.470965 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:56:00.305165 3798194 disk_info.go:125] begin check local disk info of client I1121 22:56:00.309037 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:56:00.309045 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e240 0xc00030e280] E1121 22:56:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:56:06.411975 3798194 memory.go:184] no items to output this cycle I1121 22:56:06.412135 3798194 cpu.go:275] no items to output this cycle E1121 22:56:16.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:56:16.412253 3798194 memory.go:184] no items to output this cycle I1121 22:56:16.412408 3798194 cpu.go:275] no items to output this cycle E1121 22:56:26.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:56:26.413280 3798194 memory.go:191] Add success. I1121 22:56:26.417523 3798194 cpu.go:282] Add success. I1121 22:56:26.442236 3798194 net.go:642] Add success. I1121 22:56:26.443815 3798194 net.go:778] primary dev: ETH0 I1121 22:56:26.443829 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:56:26.443842 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:56:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:56:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:56:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:56:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:56:36.411979 3798194 memory.go:184] no items to output this cycle I1121 22:56:36.412134 3798194 cpu.go:275] no items to output this cycle E1121 22:56:46.413255 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:56:46.413280 3798194 memory.go:184] no items to output this cycle I1121 22:56:46.413430 3798194 cpu.go:275] no items to output this cycle E1121 22:56:56.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:56:56.413269 3798194 memory.go:191] Add success. W1121 22:56:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:56:56.413318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:56:56.413321 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:56:56.417494 3798194 cpu.go:282] Add success. I1121 22:56:56.441899 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:56:56.442481 3798194 net.go:642] Add success. I1121 22:56:56.444051 3798194 net.go:778] primary dev: ETH0 I1121 22:56:56.444065 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:56:56.444084 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:56:57.453954 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:56:57.454183 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:56:57.454194 3798194 disk_worker.go:723] disk space is not compliant W1121 22:56:57.454198 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:56:57.467926 3798194 disk_worker.go:509] system disk:vda1 I1121 22:56:57.467970 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:56:58.446037 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:56:59.450132 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:56:59.450208 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:56:59.450241 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:56:59.470695 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:57:00.309152 3798194 disk_info.go:125] begin check local disk info of client I1121 22:57:00.313744 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:57:00.313753 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fe080 0xc0003fe0c0] E1121 22:57:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:57:06.412256 3798194 memory.go:184] no items to output this cycle I1121 22:57:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 22:57:16.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:57:16.413264 3798194 memory.go:184] no items to output this cycle I1121 22:57:16.413412 3798194 cpu.go:275] no items to output this cycle I1121 22:57:17.573044 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 22:57:17.573053 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 22:57:26.412551 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:57:26.412583 3798194 memory.go:191] Add success. I1121 22:57:26.417344 3798194 cpu.go:282] Add success. I1121 22:57:26.441764 3798194 net.go:642] Add success. I1121 22:57:26.443436 3798194 net.go:778] primary dev: ETH0 I1121 22:57:26.443451 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:57:26.443465 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:57:29.450131 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:57:29.450211 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:57:29.450238 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:57:36.412781 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:57:36.412805 3798194 memory.go:184] no items to output this cycle I1121 22:57:36.412945 3798194 cpu.go:275] no items to output this cycle E1121 22:57:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:57:46.411977 3798194 memory.go:184] no items to output this cycle I1121 22:57:46.412137 3798194 cpu.go:275] no items to output this cycle E1121 22:57:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:57:56.411952 3798194 memory.go:191] Add success. W1121 22:57:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:57:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:57:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:57:56.412349 3798194 cpu.go:282] Add success. I1121 22:57:56.441895 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:57:56.442409 3798194 net.go:642] Add success. I1121 22:57:56.446501 3798194 net.go:778] primary dev: ETH0 I1121 22:57:56.446517 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:57:56.446529 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:57:57.450831 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:57:57.450991 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:57:57.451044 3798194 disk_worker.go:723] disk space is not compliant W1121 22:57:57.451047 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:57:57.459983 3798194 disk_worker.go:509] system disk:vda1 I1121 22:57:57.460030 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:57:58.446009 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:57:59.450010 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:57:59.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:57:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:57:59.471648 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:58:00.313853 3798194 disk_info.go:125] begin check local disk info of client I1121 22:58:00.317707 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:58:00.317715 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037d440 0xc00037d480] E1121 22:58:06.412844 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:58:06.412873 3798194 memory.go:184] no items to output this cycle I1121 22:58:06.413020 3798194 cpu.go:275] no items to output this cycle E1121 22:58:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:58:16.413263 3798194 memory.go:184] no items to output this cycle I1121 22:58:16.413408 3798194 cpu.go:275] no items to output this cycle E1121 22:58:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:58:26.411955 3798194 memory.go:191] Add success. I1121 22:58:26.412329 3798194 cpu.go:282] Add success. I1121 22:58:26.441996 3798194 net.go:642] Add success. I1121 22:58:26.446283 3798194 net.go:778] primary dev: ETH0 I1121 22:58:26.446299 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:58:26.446313 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:58:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:58:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:58:29.450145 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:58:36.413174 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:58:36.413195 3798194 memory.go:184] no items to output this cycle I1121 22:58:36.413549 3798194 cpu.go:275] no items to output this cycle E1121 22:58:46.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:58:46.413272 3798194 memory.go:184] no items to output this cycle I1121 22:58:46.413420 3798194 cpu.go:275] no items to output this cycle E1121 22:58:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:58:56.413271 3798194 memory.go:191] Add success. W1121 22:58:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:58:56.413323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:58:56.413326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:58:56.418074 3798194 cpu.go:282] Add success. I1121 22:58:56.441650 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:58:56.442138 3798194 net.go:642] Add success. I1121 22:58:56.443647 3798194 net.go:778] primary dev: ETH0 I1121 22:58:56.443662 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:58:56.443676 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:58:57.446307 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:58:57.446597 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:58:57.446610 3798194 disk_worker.go:723] disk space is not compliant W1121 22:58:57.446613 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:58:57.451620 3798194 disk_worker.go:509] system disk:vda1 I1121 22:58:57.451663 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:58:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:58:59.450478 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:58:59.450566 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:58:59.450593 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:58:59.471118 3798194 disk_local_worker.go:436] Get disk info: [] I1121 22:59:00.317818 3798194 disk_info.go:125] begin check local disk info of client I1121 22:59:00.321720 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 22:59:00.321735 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fce40 0xc0003fce80] E1121 22:59:06.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:59:06.412876 3798194 memory.go:184] no items to output this cycle I1121 22:59:06.413023 3798194 cpu.go:275] no items to output this cycle E1121 22:59:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:59:16.413257 3798194 memory.go:184] no items to output this cycle I1121 22:59:16.413402 3798194 cpu.go:275] no items to output this cycle E1121 22:59:26.413145 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:59:26.413177 3798194 memory.go:191] Add success. I1121 22:59:26.417857 3798194 cpu.go:282] Add success. I1121 22:59:26.441301 3798194 net.go:642] Add success. I1121 22:59:26.442815 3798194 net.go:778] primary dev: ETH0 I1121 22:59:26.442829 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:59:26.442844 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:59:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:59:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:59:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 22:59:36.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:59:36.412879 3798194 memory.go:184] no items to output this cycle I1121 22:59:36.413029 3798194 cpu.go:275] no items to output this cycle E1121 22:59:46.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:59:46.412255 3798194 memory.go:184] no items to output this cycle I1121 22:59:46.412412 3798194 cpu.go:275] no items to output this cycle E1121 22:59:56.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 22:59:56.413267 3798194 memory.go:191] Add success. W1121 22:59:56.413296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 22:59:56.413314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 22:59:56.413318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 22:59:56.417323 3798194 cpu.go:282] Add success. I1121 22:59:56.441717 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 22:59:56.442217 3798194 net.go:642] Add success. I1121 22:59:56.443750 3798194 net.go:778] primary dev: ETH0 I1121 22:59:56.443765 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 22:59:56.443776 3798194 net.go:659] Add eni metrics success, count: 7 I1121 22:59:57.449837 3798194 custom_config.go:106] query custom config with name: gpu W1121 22:59:57.450073 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 22:59:57.450084 3798194 disk_worker.go:723] disk space is not compliant W1121 22:59:57.450087 3798194 disk_worker.go:743] disk inode is not compliant I1121 22:59:57.453646 3798194 disk_worker.go:509] system disk:vda1 I1121 22:59:57.453694 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 22:59:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 22:59:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 22:59:59.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 22:59:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 22:59:59.471652 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:00:00.325144 3798194 disk_info.go:125] begin check local disk info of client I1121 23:00:00.328976 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:00:00.328984 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000497b00 0xc000497b40] E1121 23:00:06.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:00:06.411973 3798194 memory.go:184] no items to output this cycle I1121 23:00:06.412133 3798194 cpu.go:275] no items to output this cycle E1121 23:00:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:00:16.412260 3798194 memory.go:184] no items to output this cycle I1121 23:00:16.412407 3798194 cpu.go:275] no items to output this cycle I1121 23:00:17.574090 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:00:17.574098 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:00:26.412908 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:00:26.412948 3798194 memory.go:191] Add success. I1121 23:00:26.417793 3798194 cpu.go:282] Add success. I1121 23:00:26.441297 3798194 net.go:642] Add success. I1121 23:00:26.442877 3798194 net.go:778] primary dev: ETH0 I1121 23:00:26.442891 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:00:26.442904 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:00:29.450308 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:00:29.450391 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:00:29.450420 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:00:36.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:00:36.412888 3798194 memory.go:184] no items to output this cycle I1121 23:00:36.413030 3798194 cpu.go:275] no items to output this cycle E1121 23:00:46.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:00:46.412885 3798194 memory.go:184] no items to output this cycle I1121 23:00:46.413035 3798194 cpu.go:275] no items to output this cycle E1121 23:00:56.411910 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:00:56.411942 3798194 memory.go:191] Add success. W1121 23:00:56.411968 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:00:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:00:56.411983 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:00:56.412335 3798194 cpu.go:282] Add success. I1121 23:00:56.446854 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:00:56.447377 3798194 net.go:642] Add success. I1121 23:00:56.451462 3798194 net.go:778] primary dev: ETH0 I1121 23:00:56.451478 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:00:56.451491 3798194 net.go:659] Add eni metrics success, count: 7 E1121 23:00:57.450721 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 23:00:57.450735 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:00:57.450919 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:00:57.450928 3798194 disk_worker.go:723] disk space is not compliant W1121 23:00:57.450931 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:00:57.459243 3798194 disk_worker.go:509] system disk:vda1 I1121 23:00:57.459286 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 23:00:58.448086 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 23:00:58.448100 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 23:00:59.448597 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 23:00:59.449673 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:00:59.449723 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:00:59.449744 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:00:59.471286 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:01:00.332616 3798194 disk_info.go:125] begin check local disk info of client I1121 23:01:00.336506 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:01:00.336515 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f8ec0 0xc0003f8f00] E1121 23:01:06.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:01:06.412257 3798194 memory.go:184] no items to output this cycle I1121 23:01:06.412401 3798194 cpu.go:275] no items to output this cycle E1121 23:01:16.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:01:16.413254 3798194 memory.go:184] no items to output this cycle I1121 23:01:16.413403 3798194 cpu.go:275] no items to output this cycle E1121 23:01:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:01:26.412273 3798194 memory.go:191] Add success. I1121 23:01:26.416669 3798194 cpu.go:282] Add success. I1121 23:01:26.442210 3798194 net.go:642] Add success. I1121 23:01:26.443713 3798194 net.go:778] primary dev: ETH0 I1121 23:01:26.443728 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:01:26.443741 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:01:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:01:29.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:01:29.450512 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:01:36.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:01:36.412262 3798194 memory.go:184] no items to output this cycle I1121 23:01:36.412406 3798194 cpu.go:275] no items to output this cycle E1121 23:01:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:01:46.411972 3798194 memory.go:184] no items to output this cycle I1121 23:01:46.412127 3798194 cpu.go:275] no items to output this cycle E1121 23:01:56.413011 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:01:56.413044 3798194 memory.go:191] Add success. W1121 23:01:56.413071 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:01:56.413083 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:01:56.413086 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:01:56.417582 3798194 cpu.go:282] Add success. I1121 23:01:56.440927 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:01:56.441484 3798194 net.go:642] Add success. I1121 23:01:56.442967 3798194 net.go:778] primary dev: ETH0 I1121 23:01:56.442981 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:01:56.442994 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:01:57.449211 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:01:57.449400 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:01:57.449455 3798194 disk_worker.go:723] disk space is not compliant W1121 23:01:57.449458 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:01:57.451341 3798194 disk_worker.go:509] system disk:vda1 I1121 23:01:57.451381 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:01:58.446367 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:01:59.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:01:59.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:01:59.450496 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:01:59.471007 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:02:00.339776 3798194 disk_info.go:125] begin check local disk info of client I1121 23:02:00.343695 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:02:00.343703 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002ff400 0xc0002ff440] E1121 23:02:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:02:06.411980 3798194 memory.go:184] no items to output this cycle I1121 23:02:06.412143 3798194 cpu.go:275] no items to output this cycle E1121 23:02:16.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:02:16.412251 3798194 memory.go:184] no items to output this cycle I1121 23:02:16.412406 3798194 cpu.go:275] no items to output this cycle E1121 23:02:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:02:26.411951 3798194 memory.go:191] Add success. I1121 23:02:26.412313 3798194 cpu.go:282] Add success. I1121 23:02:26.441921 3798194 net.go:642] Add success. I1121 23:02:26.446538 3798194 net.go:778] primary dev: ETH0 I1121 23:02:26.446555 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:02:26.446570 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:02:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:02:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:02:29.450495 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:02:36.413250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:02:36.413273 3798194 memory.go:184] no items to output this cycle I1121 23:02:36.413413 3798194 cpu.go:275] no items to output this cycle E1121 23:02:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:02:46.411975 3798194 memory.go:184] no items to output this cycle I1121 23:02:46.412138 3798194 cpu.go:275] no items to output this cycle E1121 23:02:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:02:56.411961 3798194 memory.go:191] Add success. W1121 23:02:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:02:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:02:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:02:56.412382 3798194 cpu.go:282] Add success. I1121 23:02:56.440915 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:02:56.441440 3798194 net.go:642] Add success. I1121 23:02:56.445477 3798194 net.go:778] primary dev: ETH0 I1121 23:02:56.445492 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:02:56.445504 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:02:57.446488 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:02:57.446741 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:02:57.446753 3798194 disk_worker.go:723] disk space is not compliant W1121 23:02:57.446756 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:02:57.449614 3798194 disk_worker.go:509] system disk:vda1 I1121 23:02:57.449654 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:02:58.446368 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:02:59.450614 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:02:59.450696 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:02:59.450725 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:02:59.471258 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:03:00.347150 3798194 disk_info.go:125] begin check local disk info of client I1121 23:03:00.350997 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:03:00.351012 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0d40 0xc0002e0d80] E1121 23:03:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:03:06.412253 3798194 memory.go:184] no items to output this cycle I1121 23:03:06.412397 3798194 cpu.go:275] no items to output this cycle E1121 23:03:16.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:03:16.413248 3798194 memory.go:184] no items to output this cycle I1121 23:03:16.413394 3798194 cpu.go:275] no items to output this cycle I1121 23:03:17.574218 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:03:17.574226 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:03:26.411203 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:03:26.411244 3798194 memory.go:191] Add success. I1121 23:03:26.411571 3798194 cpu.go:282] Add success. I1121 23:03:26.441269 3798194 net.go:642] Add success. I1121 23:03:26.445491 3798194 net.go:778] primary dev: ETH0 I1121 23:03:26.445507 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:03:26.445519 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:03:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:03:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:03:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:03:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:03:36.412257 3798194 memory.go:184] no items to output this cycle I1121 23:03:36.412404 3798194 cpu.go:275] no items to output this cycle E1121 23:03:46.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:03:46.411983 3798194 memory.go:184] no items to output this cycle I1121 23:03:46.412157 3798194 cpu.go:275] no items to output this cycle E1121 23:03:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:03:56.411952 3798194 memory.go:191] Add success. W1121 23:03:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:03:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:03:56.411993 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:03:56.412339 3798194 cpu.go:282] Add success. I1121 23:03:56.441821 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:03:56.442317 3798194 net.go:642] Add success. I1121 23:03:56.446840 3798194 net.go:778] primary dev: ETH0 I1121 23:03:56.446854 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:03:56.446867 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:03:57.450690 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:03:57.450892 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:03:57.450903 3798194 disk_worker.go:723] disk space is not compliant W1121 23:03:57.450907 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:03:57.459205 3798194 disk_worker.go:509] system disk:vda1 I1121 23:03:57.459246 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:03:58.446009 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:03:59.450162 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:03:59.450256 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:03:59.450300 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:03:59.471010 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:04:00.351123 3798194 disk_info.go:125] begin check local disk info of client I1121 23:04:00.355063 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:04:00.355073 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000476940 0xc000476980] E1121 23:04:06.412088 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:04:06.412130 3798194 memory.go:184] no items to output this cycle I1121 23:04:06.412276 3798194 cpu.go:275] no items to output this cycle E1121 23:04:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:04:16.412269 3798194 memory.go:184] no items to output this cycle I1121 23:04:16.412416 3798194 cpu.go:275] no items to output this cycle E1121 23:04:26.411929 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:04:26.411961 3798194 memory.go:191] Add success. I1121 23:04:26.412319 3798194 cpu.go:282] Add success. I1121 23:04:26.441930 3798194 net.go:642] Add success. I1121 23:04:26.446242 3798194 net.go:778] primary dev: ETH0 I1121 23:04:26.446256 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:04:26.446270 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:04:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:04:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:04:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:04:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:04:36.412260 3798194 memory.go:184] no items to output this cycle I1121 23:04:36.412399 3798194 cpu.go:275] no items to output this cycle E1121 23:04:46.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:04:46.413266 3798194 memory.go:184] no items to output this cycle I1121 23:04:46.413421 3798194 cpu.go:275] no items to output this cycle E1121 23:04:56.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:04:56.411961 3798194 memory.go:191] Add success. W1121 23:04:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:04:56.412006 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:04:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:04:56.412371 3798194 cpu.go:282] Add success. I1121 23:04:56.441836 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:04:56.442444 3798194 net.go:642] Add success. I1121 23:04:56.447069 3798194 net.go:778] primary dev: ETH0 I1121 23:04:56.447083 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:04:56.447095 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:04:57.450780 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:04:57.450956 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:04:57.451014 3798194 disk_worker.go:723] disk space is not compliant W1121 23:04:57.451018 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:04:57.460045 3798194 disk_worker.go:509] system disk:vda1 I1121 23:04:57.460088 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:04:58.446662 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:04:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:04:59.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:04:59.450508 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:04:59.471025 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:05:00.355175 3798194 disk_info.go:125] begin check local disk info of client I1121 23:05:00.359012 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:05:00.359021 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a4280 0xc0003a42c0] E1121 23:05:06.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:05:06.413254 3798194 memory.go:184] no items to output this cycle I1121 23:05:06.413397 3798194 cpu.go:275] no items to output this cycle E1121 23:05:16.411975 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:05:16.412001 3798194 memory.go:184] no items to output this cycle I1121 23:05:16.412160 3798194 cpu.go:275] no items to output this cycle E1121 23:05:26.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:05:26.411956 3798194 memory.go:191] Add success. I1121 23:05:26.412294 3798194 cpu.go:282] Add success. I1121 23:05:26.441751 3798194 net.go:642] Add success. I1121 23:05:26.445678 3798194 net.go:778] primary dev: ETH0 I1121 23:05:26.445694 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:05:26.445817 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:05:29.450034 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:05:29.450118 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:05:29.450150 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:05:36.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:05:36.411987 3798194 memory.go:184] no items to output this cycle I1121 23:05:36.412143 3798194 cpu.go:275] no items to output this cycle E1121 23:05:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:05:46.411974 3798194 memory.go:184] no items to output this cycle I1121 23:05:46.412135 3798194 cpu.go:275] no items to output this cycle E1121 23:05:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:05:56.411949 3798194 memory.go:191] Add success. W1121 23:05:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:05:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:05:56.411990 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:05:56.412337 3798194 cpu.go:282] Add success. I1121 23:05:56.441565 3798194 event_worker.go:152] Polling the log file for events... I1121 23:05:56.444805 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:05:56.445283 3798194 net.go:642] Add success. I1121 23:05:56.449471 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795271756, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 23:05:56.449591 3798194 net.go:778] primary dev: ETH0 I1121 23:05:56.449606 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:05:56.449618 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:05:56.460040 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 23:05:57.445581 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 23:05:57.445596 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:05:57.445788 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:05:57.445850 3798194 disk_worker.go:723] disk space is not compliant W1121 23:05:57.445853 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:05:57.447975 3798194 disk_worker.go:509] system disk:vda1 I1121 23:05:57.448026 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 23:05:58.448087 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 23:05:58.448102 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 23:05:59.448607 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 23:05:59.449660 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:05:59.449712 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:05:59.449736 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:05:59.471191 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:06:00.359152 3798194 disk_info.go:125] begin check local disk info of client I1121 23:06:00.363017 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:06:00.363026 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a5ac0 0xc0003a5b00] E1121 23:06:06.412789 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:06:06.412811 3798194 memory.go:184] no items to output this cycle I1121 23:06:06.412971 3798194 cpu.go:275] no items to output this cycle E1121 23:06:16.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:06:16.413259 3798194 memory.go:184] no items to output this cycle I1121 23:06:16.413415 3798194 cpu.go:275] no items to output this cycle I1121 23:06:17.574397 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:06:17.574405 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:06:26.411386 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:06:26.411420 3798194 memory.go:191] Add success. I1121 23:06:26.411763 3798194 cpu.go:282] Add success. I1121 23:06:26.441308 3798194 net.go:642] Add success. I1121 23:06:26.445427 3798194 net.go:778] primary dev: ETH0 I1121 23:06:26.445442 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:06:26.445454 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:06:29.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:06:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:06:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:06:36.412405 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:06:36.412427 3798194 memory.go:184] no items to output this cycle I1121 23:06:36.412571 3798194 cpu.go:275] no items to output this cycle E1121 23:06:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:06:46.411982 3798194 memory.go:184] no items to output this cycle I1121 23:06:46.412144 3798194 cpu.go:275] no items to output this cycle E1121 23:06:56.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:06:56.413258 3798194 memory.go:191] Add success. W1121 23:06:56.413285 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:06:56.413296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:06:56.413299 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:06:56.417826 3798194 cpu.go:282] Add success. I1121 23:06:56.441560 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:06:56.442371 3798194 net.go:642] Add success. I1121 23:06:56.444203 3798194 net.go:778] primary dev: ETH0 I1121 23:06:56.444219 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:06:56.444239 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:06:57.451000 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:06:57.451181 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:06:57.451235 3798194 disk_worker.go:723] disk space is not compliant W1121 23:06:57.451238 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:06:57.459991 3798194 disk_worker.go:509] system disk:vda1 I1121 23:06:57.460032 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:06:58.445691 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:06:59.449666 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:06:59.449744 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:06:59.449768 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:06:59.471287 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:07:00.363133 3798194 disk_info.go:125] begin check local disk info of client I1121 23:07:00.366928 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:07:00.366937 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004d3500 0xc0004d3540] E1121 23:07:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:07:06.412254 3798194 memory.go:184] no items to output this cycle I1121 23:07:06.412417 3798194 cpu.go:275] no items to output this cycle E1121 23:07:16.413256 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:07:16.413278 3798194 memory.go:184] no items to output this cycle I1121 23:07:16.413445 3798194 cpu.go:275] no items to output this cycle E1121 23:07:26.411938 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:07:26.411969 3798194 memory.go:191] Add success. I1121 23:07:26.412306 3798194 cpu.go:282] Add success. I1121 23:07:26.442187 3798194 net.go:642] Add success. I1121 23:07:26.446334 3798194 net.go:778] primary dev: ETH0 I1121 23:07:26.446350 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:07:26.446366 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:07:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:07:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:07:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:07:36.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:07:36.411984 3798194 memory.go:184] no items to output this cycle I1121 23:07:36.412144 3798194 cpu.go:275] no items to output this cycle E1121 23:07:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:07:46.413257 3798194 memory.go:184] no items to output this cycle I1121 23:07:46.413400 3798194 cpu.go:275] no items to output this cycle E1121 23:07:56.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:07:56.412262 3798194 memory.go:191] Add success. W1121 23:07:56.412296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:07:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:07:56.412311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:07:56.416421 3798194 cpu.go:282] Add success. I1121 23:07:56.441176 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:07:56.441642 3798194 net.go:642] Add success. I1121 23:07:56.443093 3798194 net.go:778] primary dev: ETH0 I1121 23:07:56.443108 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:07:56.443133 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:07:57.450938 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:07:57.451177 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:07:57.451188 3798194 disk_worker.go:723] disk space is not compliant W1121 23:07:57.451191 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:07:57.459952 3798194 disk_worker.go:509] system disk:vda1 I1121 23:07:57.459996 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:07:58.446373 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:07:59.450476 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:07:59.450556 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:07:59.450582 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:07:59.471106 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:08:00.367029 3798194 disk_info.go:125] begin check local disk info of client I1121 23:08:00.370906 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:08:00.370915 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000366980 0xc0003669c0] E1121 23:08:06.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:08:06.412881 3798194 memory.go:184] no items to output this cycle I1121 23:08:06.413023 3798194 cpu.go:275] no items to output this cycle E1121 23:08:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:08:16.411978 3798194 memory.go:184] no items to output this cycle I1121 23:08:16.412407 3798194 cpu.go:275] no items to output this cycle E1121 23:08:26.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:08:26.412888 3798194 memory.go:191] Add success. I1121 23:08:26.417102 3798194 cpu.go:282] Add success. I1121 23:08:26.441899 3798194 net.go:642] Add success. I1121 23:08:26.443515 3798194 net.go:778] primary dev: ETH0 I1121 23:08:26.443529 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:08:26.443542 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:08:29.450186 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:08:29.450271 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:08:29.450299 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:08:36.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:08:36.411974 3798194 memory.go:184] no items to output this cycle I1121 23:08:36.412126 3798194 cpu.go:275] no items to output this cycle E1121 23:08:46.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:08:46.413256 3798194 memory.go:184] no items to output this cycle I1121 23:08:46.413401 3798194 cpu.go:275] no items to output this cycle E1121 23:08:56.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:08:56.413277 3798194 memory.go:191] Add success. W1121 23:08:56.413315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:08:56.413326 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:08:56.413330 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:08:56.417566 3798194 cpu.go:282] Add success. I1121 23:08:56.440902 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:08:56.441424 3798194 net.go:642] Add success. I1121 23:08:56.442907 3798194 net.go:778] primary dev: ETH0 I1121 23:08:56.442921 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:08:56.442934 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:08:57.449591 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:08:57.449837 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:08:57.449848 3798194 disk_worker.go:723] disk space is not compliant W1121 23:08:57.449852 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:08:57.451691 3798194 disk_worker.go:509] system disk:vda1 I1121 23:08:57.451737 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:08:58.446368 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:08:59.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:08:59.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:08:59.450503 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:08:59.471047 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:09:00.373894 3798194 disk_info.go:125] begin check local disk info of client I1121 23:09:00.377771 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:09:00.377781 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00032bc40 0xc00032bc80] E1121 23:09:06.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:09:06.412885 3798194 memory.go:184] no items to output this cycle I1121 23:09:06.413029 3798194 cpu.go:275] no items to output this cycle E1121 23:09:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:09:16.413259 3798194 memory.go:184] no items to output this cycle I1121 23:09:16.413405 3798194 cpu.go:275] no items to output this cycle I1121 23:09:17.574564 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:09:17.574573 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:09:26.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:09:26.412893 3798194 memory.go:191] Add success. I1121 23:09:26.416871 3798194 cpu.go:282] Add success. I1121 23:09:26.441293 3798194 net.go:642] Add success. I1121 23:09:26.442790 3798194 net.go:778] primary dev: ETH0 I1121 23:09:26.442804 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:09:26.442816 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:09:29.450572 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:09:29.450652 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:09:29.450681 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:09:36.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:09:36.412263 3798194 memory.go:184] no items to output this cycle I1121 23:09:36.412406 3798194 cpu.go:275] no items to output this cycle E1121 23:09:46.413457 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:09:46.413478 3798194 memory.go:184] no items to output this cycle I1121 23:09:46.413624 3798194 cpu.go:275] no items to output this cycle E1121 23:09:56.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:09:56.412276 3798194 memory.go:191] Add success. W1121 23:09:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:09:56.412317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:09:56.412320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:09:56.416425 3798194 cpu.go:282] Add success. I1121 23:09:56.441823 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:09:56.442328 3798194 net.go:642] Add success. I1121 23:09:56.443722 3798194 net.go:778] primary dev: ETH0 I1121 23:09:56.443737 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:09:56.443749 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:09:57.450917 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:09:57.451130 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:09:57.451143 3798194 disk_worker.go:723] disk space is not compliant W1121 23:09:57.451147 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:09:57.460034 3798194 disk_worker.go:509] system disk:vda1 I1121 23:09:57.460074 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:09:58.446383 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:09:59.450133 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:09:59.450213 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:09:59.450238 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:09:59.470729 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:10:00.382316 3798194 disk_info.go:125] begin check local disk info of client I1121 23:10:00.386208 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:10:00.386221 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e1580 0xc0002e15c0] E1121 23:10:06.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:10:06.412879 3798194 memory.go:184] no items to output this cycle I1121 23:10:06.413026 3798194 cpu.go:275] no items to output this cycle E1121 23:10:16.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:10:16.412259 3798194 memory.go:184] no items to output this cycle I1121 23:10:16.412412 3798194 cpu.go:275] no items to output this cycle E1121 23:10:26.412842 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:10:26.412875 3798194 memory.go:191] Add success. I1121 23:10:26.417254 3798194 cpu.go:282] Add success. I1121 23:10:26.441672 3798194 net.go:642] Add success. I1121 23:10:26.443021 3798194 net.go:778] primary dev: ETH0 I1121 23:10:26.443035 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:10:26.443047 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:10:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:10:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:10:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:10:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:10:36.411982 3798194 memory.go:184] no items to output this cycle I1121 23:10:36.412135 3798194 cpu.go:275] no items to output this cycle E1121 23:10:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:10:46.411977 3798194 memory.go:184] no items to output this cycle I1121 23:10:46.412134 3798194 cpu.go:275] no items to output this cycle E1121 23:10:56.412051 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:10:56.412083 3798194 memory.go:191] Add success. W1121 23:10:56.412129 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:10:56.412146 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:10:56.412149 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:10:56.416170 3798194 cpu.go:282] Add success. I1121 23:10:56.444265 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:10:56.444742 3798194 net.go:642] Add success. I1121 23:10:56.446297 3798194 net.go:778] primary dev: ETH0 I1121 23:10:56.446311 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:10:56.446323 3798194 net.go:659] Add eni metrics success, count: 7 E1121 23:10:57.450506 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 23:10:57.450521 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:10:57.450713 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:10:57.450722 3798194 disk_worker.go:723] disk space is not compliant W1121 23:10:57.450725 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:10:57.459072 3798194 disk_worker.go:509] system disk:vda1 I1121 23:10:57.459126 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 23:10:58.448090 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 23:10:58.448106 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 23:10:59.448627 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 23:10:59.449702 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:10:59.449760 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:10:59.449780 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:10:59.471331 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:11:00.389955 3798194 disk_info.go:125] begin check local disk info of client I1121 23:11:00.393795 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:11:00.393804 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004782c0 0xc000478300] E1121 23:11:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:11:06.411983 3798194 memory.go:184] no items to output this cycle I1121 23:11:06.412143 3798194 cpu.go:275] no items to output this cycle E1121 23:11:16.412291 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:11:16.412315 3798194 memory.go:184] no items to output this cycle I1121 23:11:16.412462 3798194 cpu.go:275] no items to output this cycle E1121 23:11:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:11:26.413264 3798194 memory.go:191] Add success. I1121 23:11:26.417435 3798194 cpu.go:282] Add success. I1121 23:11:26.441837 3798194 net.go:642] Add success. I1121 23:11:26.443231 3798194 net.go:778] primary dev: ETH0 I1121 23:11:26.443246 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:11:26.443259 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:11:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:11:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:11:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:11:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:11:36.411977 3798194 memory.go:184] no items to output this cycle I1121 23:11:36.412132 3798194 cpu.go:275] no items to output this cycle E1121 23:11:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:11:46.411978 3798194 memory.go:184] no items to output this cycle I1121 23:11:46.412141 3798194 cpu.go:275] no items to output this cycle E1121 23:11:56.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:11:56.411958 3798194 memory.go:191] Add success. W1121 23:11:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:11:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:11:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:11:56.412353 3798194 cpu.go:282] Add success. I1121 23:11:56.441826 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:11:56.442324 3798194 net.go:642] Add success. I1121 23:11:56.446258 3798194 net.go:778] primary dev: ETH0 I1121 23:11:56.446273 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:11:56.446285 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:11:57.450626 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:11:57.450822 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:11:57.450832 3798194 disk_worker.go:723] disk space is not compliant W1121 23:11:57.450835 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:11:57.459184 3798194 disk_worker.go:509] system disk:vda1 I1121 23:11:57.459226 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:11:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:11:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:11:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:11:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:11:59.471656 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:12:00.398135 3798194 disk_info.go:125] begin check local disk info of client I1121 23:12:00.402069 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:12:00.402080 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037d900 0xc00037d940] E1121 23:12:06.412780 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:12:06.412802 3798194 memory.go:184] no items to output this cycle I1121 23:12:06.412955 3798194 cpu.go:275] no items to output this cycle E1121 23:12:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:12:16.411983 3798194 memory.go:184] no items to output this cycle I1121 23:12:16.412145 3798194 cpu.go:275] no items to output this cycle I1121 23:12:17.575710 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:12:17.575719 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:12:26.412253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:12:26.412286 3798194 memory.go:191] Add success. I1121 23:12:26.416554 3798194 cpu.go:282] Add success. I1121 23:12:26.442098 3798194 net.go:642] Add success. I1121 23:12:26.443547 3798194 net.go:778] primary dev: ETH0 I1121 23:12:26.443561 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:12:26.443577 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:12:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:12:29.450124 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:12:29.450198 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:12:36.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:12:36.411971 3798194 memory.go:184] no items to output this cycle I1121 23:12:36.412123 3798194 cpu.go:275] no items to output this cycle E1121 23:12:46.412841 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:12:46.412864 3798194 memory.go:184] no items to output this cycle I1121 23:12:46.413007 3798194 cpu.go:275] no items to output this cycle E1121 23:12:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:12:56.411951 3798194 memory.go:191] Add success. W1121 23:12:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:12:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:12:56.412007 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:12:56.412360 3798194 cpu.go:282] Add success. I1121 23:12:56.441819 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:12:56.442412 3798194 net.go:642] Add success. I1121 23:12:56.446546 3798194 net.go:778] primary dev: ETH0 I1121 23:12:56.446561 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:12:56.446574 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:12:57.450691 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:12:57.450896 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:12:57.450906 3798194 disk_worker.go:723] disk space is not compliant W1121 23:12:57.450910 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:12:57.459330 3798194 disk_worker.go:509] system disk:vda1 I1121 23:12:57.459375 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:12:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:12:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:12:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:12:59.450151 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:12:59.471694 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:13:00.402181 3798194 disk_info.go:125] begin check local disk info of client I1121 23:13:00.406067 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:13:00.406076 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000463cc0 0xc000463d00] E1121 23:13:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:13:06.412260 3798194 memory.go:184] no items to output this cycle I1121 23:13:06.412412 3798194 cpu.go:275] no items to output this cycle E1121 23:13:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:13:16.411971 3798194 memory.go:184] no items to output this cycle I1121 23:13:16.412132 3798194 cpu.go:275] no items to output this cycle E1121 23:13:26.412789 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:13:26.412821 3798194 memory.go:191] Add success. I1121 23:13:26.417188 3798194 cpu.go:282] Add success. I1121 23:13:26.441692 3798194 net.go:642] Add success. I1121 23:13:26.443269 3798194 net.go:778] primary dev: ETH0 I1121 23:13:26.443284 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:13:26.443297 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:13:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:13:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:13:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:13:36.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:13:36.411975 3798194 memory.go:184] no items to output this cycle I1121 23:13:36.412135 3798194 cpu.go:275] no items to output this cycle E1121 23:13:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:13:46.412257 3798194 memory.go:184] no items to output this cycle I1121 23:13:46.412402 3798194 cpu.go:275] no items to output this cycle E1121 23:13:56.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:13:56.412274 3798194 memory.go:191] Add success. W1121 23:13:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:13:56.412320 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:13:56.412323 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:13:56.416624 3798194 cpu.go:282] Add success. I1121 23:13:56.440954 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:13:56.441547 3798194 net.go:642] Add success. I1121 23:13:56.443102 3798194 net.go:778] primary dev: ETH0 I1121 23:13:56.443131 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:13:56.443146 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:13:57.451216 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:13:57.451562 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:13:57.451706 3798194 disk_worker.go:723] disk space is not compliant W1121 23:13:57.451709 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:13:57.460873 3798194 disk_worker.go:509] system disk:vda1 I1121 23:13:57.460914 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:13:58.446012 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:13:59.450627 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:13:59.450697 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:13:59.450722 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:13:59.471188 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:14:00.409915 3798194 disk_info.go:125] begin check local disk info of client I1121 23:14:00.413857 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:14:00.413868 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004cf9c0 0xc0004cfa00] E1121 23:14:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:14:06.411976 3798194 memory.go:184] no items to output this cycle I1121 23:14:06.412133 3798194 cpu.go:275] no items to output this cycle E1121 23:14:16.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:14:16.413271 3798194 memory.go:184] no items to output this cycle I1121 23:14:16.413415 3798194 cpu.go:275] no items to output this cycle E1121 23:14:26.412498 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:14:26.412535 3798194 memory.go:191] Add success. I1121 23:14:26.416665 3798194 cpu.go:282] Add success. I1121 23:14:26.442201 3798194 net.go:642] Add success. I1121 23:14:26.443652 3798194 net.go:778] primary dev: ETH0 I1121 23:14:26.443667 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:14:26.443681 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:14:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:14:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:14:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:14:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:14:36.412253 3798194 memory.go:184] no items to output this cycle I1121 23:14:36.412395 3798194 cpu.go:275] no items to output this cycle E1121 23:14:46.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:14:46.413251 3798194 memory.go:184] no items to output this cycle I1121 23:14:46.413397 3798194 cpu.go:275] no items to output this cycle E1121 23:14:56.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:14:56.412274 3798194 memory.go:191] Add success. W1121 23:14:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:14:56.412315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:14:56.412318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:14:56.416508 3798194 cpu.go:282] Add success. I1121 23:14:56.441891 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:14:56.442394 3798194 net.go:642] Add success. I1121 23:14:56.443890 3798194 net.go:778] primary dev: ETH0 I1121 23:14:56.443905 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:14:56.443919 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:14:57.450577 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:14:57.450773 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:14:57.450785 3798194 disk_worker.go:723] disk space is not compliant W1121 23:14:57.450801 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:14:57.459038 3798194 disk_worker.go:509] system disk:vda1 I1121 23:14:57.459080 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:14:58.446012 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:14:59.450118 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:14:59.450194 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:14:59.450219 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:14:59.470679 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:15:00.414861 3798194 disk_info.go:125] begin check local disk info of client I1121 23:15:00.418792 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:15:00.418801 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d6c00 0xc0003d6c40] E1121 23:15:06.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:15:06.412265 3798194 memory.go:184] no items to output this cycle I1121 23:15:06.412410 3798194 cpu.go:275] no items to output this cycle E1121 23:15:16.412627 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:15:16.412649 3798194 memory.go:184] no items to output this cycle I1121 23:15:16.412810 3798194 cpu.go:275] no items to output this cycle I1121 23:15:17.575878 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:15:17.575887 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:15:26.412920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:15:26.412966 3798194 memory.go:191] Add success. I1121 23:15:26.417867 3798194 cpu.go:282] Add success. I1121 23:15:26.441515 3798194 net.go:642] Add success. I1121 23:15:26.443131 3798194 net.go:778] primary dev: ETH0 I1121 23:15:26.443152 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:15:26.443173 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:15:29.450131 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:15:29.450207 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:15:29.450235 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:15:36.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:15:36.412873 3798194 memory.go:184] no items to output this cycle I1121 23:15:36.413019 3798194 cpu.go:275] no items to output this cycle E1121 23:15:46.412957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:15:46.412979 3798194 memory.go:184] no items to output this cycle I1121 23:15:46.413141 3798194 cpu.go:275] no items to output this cycle E1121 23:15:56.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:15:56.411964 3798194 memory.go:191] Add success. W1121 23:15:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:15:56.412010 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:15:56.412013 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:15:56.412395 3798194 cpu.go:282] Add success. I1121 23:15:56.441684 3798194 event_worker.go:152] Polling the log file for events... I1121 23:15:56.445205 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:15:56.445712 3798194 net.go:642] Add success. I1121 23:15:56.449981 3798194 net.go:778] primary dev: ETH0 I1121 23:15:56.449997 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:15:56.450009 3798194 net.go:659] Add eni metrics success, count: 7 E1121 23:15:57.450639 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 23:15:57.450654 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:15:57.450855 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:15:57.450864 3798194 disk_worker.go:723] disk space is not compliant W1121 23:15:57.450867 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:15:57.459232 3798194 disk_worker.go:509] system disk:vda1 I1121 23:15:57.459277 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 23:15:58.448144 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 23:15:58.448159 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:15:59.449908 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:15:59.449961 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:15:59.449979 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:15:59.450130 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 23:15:59.471551 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:16:00.419855 3798194 disk_info.go:125] begin check local disk info of client I1121 23:16:00.423776 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:16:00.423786 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000272dc0 0xc000272e00] E1121 23:16:06.413060 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:16:06.413084 3798194 memory.go:184] no items to output this cycle I1121 23:16:06.413249 3798194 cpu.go:275] no items to output this cycle E1121 23:16:16.413155 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:16:16.413177 3798194 memory.go:184] no items to output this cycle I1121 23:16:16.413321 3798194 cpu.go:275] no items to output this cycle E1121 23:16:26.412821 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:16:26.412859 3798194 memory.go:191] Add success. I1121 23:16:26.416925 3798194 cpu.go:282] Add success. I1121 23:16:26.441471 3798194 net.go:642] Add success. I1121 23:16:26.442810 3798194 net.go:778] primary dev: ETH0 I1121 23:16:26.442825 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:16:26.442840 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:16:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:16:29.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:16:29.450511 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:16:36.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:16:36.413252 3798194 memory.go:184] no items to output this cycle I1121 23:16:36.413393 3798194 cpu.go:275] no items to output this cycle E1121 23:16:46.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:16:46.412880 3798194 memory.go:184] no items to output this cycle I1121 23:16:46.413029 3798194 cpu.go:275] no items to output this cycle E1121 23:16:56.412973 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:16:56.413011 3798194 memory.go:191] Add success. W1121 23:16:56.413041 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:16:56.413053 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:16:56.413056 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:16:56.417628 3798194 cpu.go:282] Add success. I1121 23:16:56.441273 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:16:56.441753 3798194 net.go:642] Add success. I1121 23:16:56.443182 3798194 net.go:778] primary dev: ETH0 I1121 23:16:56.443197 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:16:56.443210 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:16:57.450579 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:16:57.450751 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:16:57.450804 3798194 disk_worker.go:723] disk space is not compliant W1121 23:16:57.450807 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:16:57.459084 3798194 disk_worker.go:509] system disk:vda1 I1121 23:16:57.459143 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:16:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:16:59.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:16:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:16:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:16:59.471662 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:17:00.423845 3798194 disk_info.go:125] begin check local disk info of client I1121 23:17:00.427745 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:17:00.427755 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e600 0xc00030e640] E1121 23:17:06.412789 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:17:06.412812 3798194 memory.go:184] no items to output this cycle I1121 23:17:06.412953 3798194 cpu.go:275] no items to output this cycle E1121 23:17:16.413332 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:17:16.413357 3798194 memory.go:184] no items to output this cycle I1121 23:17:16.413509 3798194 cpu.go:275] no items to output this cycle E1121 23:17:26.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:17:26.413276 3798194 memory.go:191] Add success. I1121 23:17:26.417223 3798194 cpu.go:282] Add success. I1121 23:17:26.441757 3798194 net.go:642] Add success. I1121 23:17:26.443204 3798194 net.go:778] primary dev: ETH0 I1121 23:17:26.443219 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:17:26.443232 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:17:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:17:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:17:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:17:36.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:17:36.413268 3798194 memory.go:184] no items to output this cycle I1121 23:17:36.413415 3798194 cpu.go:275] no items to output this cycle E1121 23:17:46.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:17:46.412266 3798194 memory.go:184] no items to output this cycle I1121 23:17:46.412409 3798194 cpu.go:275] no items to output this cycle E1121 23:17:56.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:17:56.412278 3798194 memory.go:191] Add success. W1121 23:17:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:17:56.412316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:17:56.412320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:17:56.416241 3798194 cpu.go:282] Add success. I1121 23:17:56.441783 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:17:56.442300 3798194 net.go:642] Add success. I1121 23:17:56.443725 3798194 net.go:778] primary dev: ETH0 I1121 23:17:56.443740 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:17:56.443753 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:17:57.450623 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:17:57.450821 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:17:57.450831 3798194 disk_worker.go:723] disk space is not compliant W1121 23:17:57.450834 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:17:57.459055 3798194 disk_worker.go:509] system disk:vda1 I1121 23:17:57.459121 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:17:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:17:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:17:59.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:17:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:17:59.471642 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:18:00.428311 3798194 disk_info.go:125] begin check local disk info of client I1121 23:18:00.432254 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:18:00.432263 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000490040 0xc000490080] E1121 23:18:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:18:06.412260 3798194 memory.go:184] no items to output this cycle I1121 23:18:06.412402 3798194 cpu.go:275] no items to output this cycle E1121 23:18:16.412246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:18:16.412269 3798194 memory.go:184] no items to output this cycle I1121 23:18:16.412412 3798194 cpu.go:275] no items to output this cycle I1121 23:18:17.576213 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:18:17.576222 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:18:26.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:18:26.412265 3798194 memory.go:191] Add success. I1121 23:18:26.416474 3798194 cpu.go:282] Add success. I1121 23:18:26.442282 3798194 net.go:642] Add success. I1121 23:18:26.443726 3798194 net.go:778] primary dev: ETH0 I1121 23:18:26.443741 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:18:26.443755 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:18:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:18:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:18:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:18:36.412654 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:18:36.412678 3798194 memory.go:184] no items to output this cycle I1121 23:18:36.412816 3798194 cpu.go:275] no items to output this cycle E1121 23:18:46.412868 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:18:46.412892 3798194 memory.go:184] no items to output this cycle I1121 23:18:46.413040 3798194 cpu.go:275] no items to output this cycle E1121 23:18:56.413536 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:18:56.413581 3798194 memory.go:191] Add success. W1121 23:18:56.413615 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:18:56.413629 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:18:56.413635 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:18:56.418035 3798194 cpu.go:282] Add success. I1121 23:18:56.441411 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:18:56.441883 3798194 net.go:642] Add success. I1121 23:18:56.443366 3798194 net.go:778] primary dev: ETH0 I1121 23:18:56.443381 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:18:56.443393 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:18:57.446442 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:18:57.446692 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:18:57.446704 3798194 disk_worker.go:723] disk space is not compliant W1121 23:18:57.446707 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:18:57.449503 3798194 disk_worker.go:509] system disk:vda1 I1121 23:18:57.449542 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:18:58.446455 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:18:59.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:18:59.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:18:59.450494 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:18:59.471004 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:19:00.432555 3798194 disk_info.go:125] begin check local disk info of client I1121 23:19:00.436469 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:19:00.436479 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00038f480 0xc00038f4c0] E1121 23:19:06.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:19:06.412876 3798194 memory.go:184] no items to output this cycle I1121 23:19:06.413021 3798194 cpu.go:275] no items to output this cycle E1121 23:19:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:19:16.411974 3798194 memory.go:184] no items to output this cycle I1121 23:19:16.412134 3798194 cpu.go:275] no items to output this cycle E1121 23:19:26.411908 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:19:26.411939 3798194 memory.go:191] Add success. I1121 23:19:26.412297 3798194 cpu.go:282] Add success. I1121 23:19:26.441747 3798194 net.go:642] Add success. I1121 23:19:26.446143 3798194 net.go:778] primary dev: ETH0 I1121 23:19:26.446160 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:19:26.446173 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:19:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:19:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:19:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:19:36.413339 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:19:36.413428 3798194 memory.go:184] no items to output this cycle I1121 23:19:36.413780 3798194 cpu.go:275] no items to output this cycle E1121 23:19:46.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:19:46.413266 3798194 memory.go:184] no items to output this cycle I1121 23:19:46.413430 3798194 cpu.go:275] no items to output this cycle E1121 23:19:56.413089 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:19:56.413140 3798194 memory.go:191] Add success. W1121 23:19:56.413168 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:19:56.413179 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:19:56.413183 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:19:56.417346 3798194 cpu.go:282] Add success. I1121 23:19:56.441924 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:19:56.442520 3798194 net.go:642] Add success. I1121 23:19:56.444180 3798194 net.go:778] primary dev: ETH0 I1121 23:19:56.444197 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:19:56.444210 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:19:57.450496 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:19:57.450653 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:19:57.450703 3798194 disk_worker.go:723] disk space is not compliant W1121 23:19:57.450706 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:19:57.459039 3798194 disk_worker.go:509] system disk:vda1 I1121 23:19:57.459082 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:19:58.446535 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:19:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:19:59.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:19:59.450500 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:19:59.471031 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:20:00.436954 3798194 disk_info.go:125] begin check local disk info of client I1121 23:20:00.440875 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:20:00.440884 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f6e40 0xc0003f6e80] E1121 23:20:06.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:20:06.413255 3798194 memory.go:184] no items to output this cycle I1121 23:20:06.413399 3798194 cpu.go:275] no items to output this cycle E1121 23:20:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:20:16.411979 3798194 memory.go:184] no items to output this cycle I1121 23:20:16.412136 3798194 cpu.go:275] no items to output this cycle E1121 23:20:26.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:20:26.412892 3798194 memory.go:191] Add success. I1121 23:20:26.417035 3798194 cpu.go:282] Add success. I1121 23:20:26.441643 3798194 net.go:642] Add success. I1121 23:20:26.443163 3798194 net.go:778] primary dev: ETH0 I1121 23:20:26.443179 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:20:26.443194 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:20:29.450007 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:20:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:20:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:20:36.411404 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:20:36.411428 3798194 memory.go:184] no items to output this cycle I1121 23:20:36.411574 3798194 cpu.go:275] no items to output this cycle E1121 23:20:46.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:20:46.412259 3798194 memory.go:184] no items to output this cycle I1121 23:20:46.412409 3798194 cpu.go:275] no items to output this cycle E1121 23:20:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:20:56.411947 3798194 memory.go:191] Add success. W1121 23:20:56.411973 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:20:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:20:56.411988 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:20:56.412346 3798194 cpu.go:282] Add success. I1121 23:20:56.444631 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:20:56.445107 3798194 net.go:642] Add success. I1121 23:20:56.449263 3798194 net.go:778] primary dev: ETH0 I1121 23:20:56.449277 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:20:56.449289 3798194 net.go:659] Add eni metrics success, count: 7 E1121 23:20:57.445594 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 23:20:57.445619 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:20:57.445806 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:20:57.445868 3798194 disk_worker.go:723] disk space is not compliant W1121 23:20:57.445872 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:20:57.450863 3798194 disk_worker.go:509] system disk:vda1 I1121 23:20:57.450903 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 23:20:58.448147 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 23:20:58.448163 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 23:20:59.448641 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 23:20:59.449719 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:20:59.449772 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:20:59.449801 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:20:59.471353 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:21:00.440936 3798194 disk_info.go:125] begin check local disk info of client I1121 23:21:00.444866 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:21:00.444876 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000499ec0 0xc000499f00] E1121 23:21:06.412759 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:21:06.412783 3798194 memory.go:184] no items to output this cycle I1121 23:21:06.412952 3798194 cpu.go:275] no items to output this cycle E1121 23:21:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:21:16.411980 3798194 memory.go:184] no items to output this cycle I1121 23:21:16.412138 3798194 cpu.go:275] no items to output this cycle I1121 23:21:17.576374 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:21:17.576382 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:21:26.412838 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:21:26.412873 3798194 memory.go:191] Add success. I1121 23:21:26.417151 3798194 cpu.go:282] Add success. I1121 23:21:26.441658 3798194 net.go:642] Add success. I1121 23:21:26.443215 3798194 net.go:778] primary dev: ETH0 I1121 23:21:26.443231 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:21:26.443246 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:21:29.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:21:29.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:21:29.450493 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:21:36.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:21:36.412252 3798194 memory.go:184] no items to output this cycle I1121 23:21:36.412393 3798194 cpu.go:275] no items to output this cycle E1121 23:21:46.413251 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:21:46.413272 3798194 memory.go:184] no items to output this cycle I1121 23:21:46.413420 3798194 cpu.go:275] no items to output this cycle E1121 23:21:56.412250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:21:56.412284 3798194 memory.go:191] Add success. W1121 23:21:56.412316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:21:56.412330 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:21:56.412333 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:21:56.416514 3798194 cpu.go:282] Add success. I1121 23:21:56.441071 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:21:56.441558 3798194 net.go:642] Add success. I1121 23:21:56.443105 3798194 net.go:778] primary dev: ETH0 I1121 23:21:56.443131 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:21:56.443144 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:21:57.451312 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:21:57.451517 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:21:57.451529 3798194 disk_worker.go:723] disk space is not compliant W1121 23:21:57.451533 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:21:57.459964 3798194 disk_worker.go:509] system disk:vda1 I1121 23:21:57.460007 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:21:58.446204 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:21:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:21:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:21:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:21:59.471670 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:22:00.444955 3798194 disk_info.go:125] begin check local disk info of client I1121 23:22:00.448875 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:22:00.448884 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000409180 0xc0004091c0] E1121 23:22:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:22:06.411977 3798194 memory.go:184] no items to output this cycle I1121 23:22:06.412144 3798194 cpu.go:275] no items to output this cycle E1121 23:22:16.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:22:16.413254 3798194 memory.go:184] no items to output this cycle I1121 23:22:16.413402 3798194 cpu.go:275] no items to output this cycle E1121 23:22:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:22:26.413274 3798194 memory.go:191] Add success. I1121 23:22:26.417269 3798194 cpu.go:282] Add success. I1121 23:22:26.441760 3798194 net.go:642] Add success. I1121 23:22:26.443310 3798194 net.go:778] primary dev: ETH0 I1121 23:22:26.443325 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:22:26.443338 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:22:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:22:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:22:29.450154 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:22:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:22:36.411976 3798194 memory.go:184] no items to output this cycle I1121 23:22:36.412131 3798194 cpu.go:275] no items to output this cycle E1121 23:22:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:22:46.411984 3798194 memory.go:184] no items to output this cycle I1121 23:22:46.412143 3798194 cpu.go:275] no items to output this cycle E1121 23:22:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:22:56.411948 3798194 memory.go:191] Add success. W1121 23:22:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:22:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:22:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:22:56.412357 3798194 cpu.go:282] Add success. I1121 23:22:56.441826 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:22:56.442399 3798194 net.go:642] Add success. I1121 23:22:56.446240 3798194 net.go:778] primary dev: ETH0 I1121 23:22:56.446255 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:22:56.446266 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:22:57.451096 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:22:57.451276 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:22:57.451331 3798194 disk_worker.go:723] disk space is not compliant W1121 23:22:57.451334 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:22:57.462014 3798194 disk_worker.go:509] system disk:vda1 I1121 23:22:57.462057 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:22:58.446016 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:22:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:22:59.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:22:59.450144 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:22:59.470779 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:23:00.448992 3798194 disk_info.go:125] begin check local disk info of client I1121 23:23:00.452975 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:23:00.452983 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002fd440 0xc0002fd480] E1121 23:23:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:23:06.412254 3798194 memory.go:184] no items to output this cycle I1121 23:23:06.412398 3798194 cpu.go:275] no items to output this cycle E1121 23:23:16.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:23:16.412885 3798194 memory.go:184] no items to output this cycle I1121 23:23:16.413029 3798194 cpu.go:275] no items to output this cycle E1121 23:23:26.413502 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:23:26.413534 3798194 memory.go:191] Add success. I1121 23:23:26.417622 3798194 cpu.go:282] Add success. I1121 23:23:26.441266 3798194 net.go:642] Add success. I1121 23:23:26.442758 3798194 net.go:778] primary dev: ETH0 I1121 23:23:26.442773 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:23:26.442788 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:23:29.449677 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:23:29.449756 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:23:29.449781 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:23:36.412867 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:23:36.412888 3798194 memory.go:184] no items to output this cycle I1121 23:23:36.413028 3798194 cpu.go:275] no items to output this cycle E1121 23:23:46.413303 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:23:46.413335 3798194 memory.go:184] no items to output this cycle I1121 23:23:46.413534 3798194 cpu.go:275] no items to output this cycle E1121 23:23:56.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:23:56.412283 3798194 memory.go:191] Add success. W1121 23:23:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:23:56.412323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:23:56.412326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:23:56.416605 3798194 cpu.go:282] Add success. I1121 23:23:56.440959 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:23:56.441564 3798194 net.go:642] Add success. I1121 23:23:56.443166 3798194 net.go:778] primary dev: ETH0 I1121 23:23:56.443180 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:23:56.443193 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:23:57.450920 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:23:57.451078 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:23:57.451152 3798194 disk_worker.go:723] disk space is not compliant W1121 23:23:57.451155 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:23:57.459783 3798194 disk_worker.go:509] system disk:vda1 I1121 23:23:57.459828 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:23:58.446371 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:23:59.450386 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:23:59.450461 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:23:59.450489 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:23:59.471085 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:24:00.453415 3798194 disk_info.go:125] begin check local disk info of client I1121 23:24:00.457367 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:24:00.457379 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0d80 0xc0002e0dc0] E1121 23:24:06.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:24:06.413258 3798194 memory.go:184] no items to output this cycle I1121 23:24:06.413398 3798194 cpu.go:275] no items to output this cycle E1121 23:24:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:24:16.413259 3798194 memory.go:184] no items to output this cycle I1121 23:24:16.413415 3798194 cpu.go:275] no items to output this cycle I1121 23:24:17.576540 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:24:17.576549 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:24:26.413300 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:24:26.413336 3798194 memory.go:191] Add success. I1121 23:24:26.417414 3798194 cpu.go:282] Add success. I1121 23:24:26.441997 3798194 net.go:642] Add success. I1121 23:24:26.443508 3798194 net.go:778] primary dev: ETH0 I1121 23:24:26.443524 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:24:26.443537 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:24:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:24:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:24:29.450519 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:24:36.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:24:36.412266 3798194 memory.go:184] no items to output this cycle I1121 23:24:36.412408 3798194 cpu.go:275] no items to output this cycle E1121 23:24:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:24:46.411985 3798194 memory.go:184] no items to output this cycle I1121 23:24:46.412141 3798194 cpu.go:275] no items to output this cycle E1121 23:24:56.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:24:56.412269 3798194 memory.go:191] Add success. W1121 23:24:56.412296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:24:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:24:56.412311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:24:56.417152 3798194 cpu.go:282] Add success. I1121 23:24:56.441600 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:24:56.442092 3798194 net.go:642] Add success. I1121 23:24:56.443622 3798194 net.go:778] primary dev: ETH0 I1121 23:24:56.443636 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:24:56.443649 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:24:57.450809 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:24:57.451021 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:24:57.451032 3798194 disk_worker.go:723] disk space is not compliant W1121 23:24:57.451035 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:24:57.459786 3798194 disk_worker.go:509] system disk:vda1 I1121 23:24:57.459839 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:24:58.446309 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:24:59.450414 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:24:59.450494 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:24:59.450524 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:24:59.471063 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:25:00.458071 3798194 disk_info.go:125] begin check local disk info of client I1121 23:25:00.461932 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:25:00.461941 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046c800 0xc00046c840] E1121 23:25:06.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:25:06.412876 3798194 memory.go:184] no items to output this cycle I1121 23:25:06.413019 3798194 cpu.go:275] no items to output this cycle E1121 23:25:16.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:25:16.412253 3798194 memory.go:184] no items to output this cycle I1121 23:25:16.412401 3798194 cpu.go:275] no items to output this cycle E1121 23:25:26.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:25:26.411949 3798194 memory.go:191] Add success. I1121 23:25:26.412285 3798194 cpu.go:282] Add success. I1121 23:25:26.441758 3798194 net.go:642] Add success. I1121 23:25:26.445605 3798194 net.go:778] primary dev: ETH0 I1121 23:25:26.445621 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:25:26.445635 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:25:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:25:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:25:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:25:36.412421 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:25:36.412444 3798194 memory.go:184] no items to output this cycle I1121 23:25:36.412595 3798194 cpu.go:275] no items to output this cycle E1121 23:25:46.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:25:46.411990 3798194 memory.go:184] no items to output this cycle I1121 23:25:46.412151 3798194 cpu.go:275] no items to output this cycle E1121 23:25:56.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:25:56.413267 3798194 memory.go:191] Add success. W1121 23:25:56.413302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:25:56.413314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:25:56.413317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:25:56.417786 3798194 cpu.go:282] Add success. I1121 23:25:56.443830 3798194 event_worker.go:152] Polling the log file for events... I1121 23:25:56.444048 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:25:56.444550 3798194 net.go:642] Add success. I1121 23:25:56.446010 3798194 net.go:778] primary dev: ETH0 I1121 23:25:56.446026 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:25:56.446039 3798194 net.go:659] Add eni metrics success, count: 7 W1121 23:25:57.443973 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:25:57.443988 3798194 disk_worker.go:723] disk space is not compliant W1121 23:25:57.443992 3798194 disk_worker.go:743] disk inode is not compliant E1121 23:25:57.452304 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 23:25:57.452316 3798194 custom_config.go:106] query custom config with name: gpu I1121 23:25:57.452339 3798194 disk_worker.go:509] system disk:vda1 I1121 23:25:57.452379 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 23:25:58.448424 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 23:25:58.448440 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 23:25:59.448614 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 23:25:59.449691 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:25:59.449742 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:25:59.449762 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:25:59.471277 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:26:00.462029 3798194 disk_info.go:125] begin check local disk info of client I1121 23:26:00.465939 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:26:00.465949 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000394240 0xc000394280] E1121 23:26:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:26:06.411983 3798194 memory.go:184] no items to output this cycle I1121 23:26:06.412134 3798194 cpu.go:275] no items to output this cycle E1121 23:26:16.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:26:16.413266 3798194 memory.go:184] no items to output this cycle I1121 23:26:16.413408 3798194 cpu.go:275] no items to output this cycle E1121 23:26:26.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:26:26.412888 3798194 memory.go:191] Add success. I1121 23:26:26.417233 3798194 cpu.go:282] Add success. I1121 23:26:26.441756 3798194 net.go:642] Add success. I1121 23:26:26.443105 3798194 net.go:778] primary dev: ETH0 I1121 23:26:26.443131 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:26:26.443144 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:26:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:26:29.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:26:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:26:36.412985 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:26:36.413073 3798194 memory.go:184] no items to output this cycle I1121 23:26:36.413488 3798194 cpu.go:275] no items to output this cycle E1121 23:26:46.412001 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:26:46.412025 3798194 memory.go:184] no items to output this cycle I1121 23:26:46.412195 3798194 cpu.go:275] no items to output this cycle E1121 23:26:56.412554 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:26:56.412587 3798194 memory.go:191] Add success. W1121 23:26:56.412616 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:26:56.412627 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:26:56.412631 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:26:56.417045 3798194 cpu.go:282] Add success. I1121 23:26:56.441817 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:26:56.442472 3798194 net.go:642] Add success. I1121 23:26:56.443872 3798194 net.go:778] primary dev: ETH0 I1121 23:26:56.443888 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:26:56.443901 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:26:57.449329 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:26:57.449524 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:26:57.449581 3798194 disk_worker.go:723] disk space is not compliant W1121 23:26:57.449585 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:26:57.451394 3798194 disk_worker.go:509] system disk:vda1 I1121 23:26:57.451433 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:26:58.445987 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:26:59.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:26:59.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:26:59.450503 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:26:59.471038 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:27:00.466714 3798194 disk_info.go:125] begin check local disk info of client I1121 23:27:00.470620 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:27:00.470628 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000446640 0xc000446680] E1121 23:27:06.411938 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:27:06.411963 3798194 memory.go:184] no items to output this cycle I1121 23:27:06.412105 3798194 cpu.go:275] no items to output this cycle E1121 23:27:16.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:27:16.412259 3798194 memory.go:184] no items to output this cycle I1121 23:27:16.412403 3798194 cpu.go:275] no items to output this cycle I1121 23:27:17.576708 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:27:17.576716 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:27:26.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:27:26.413275 3798194 memory.go:191] Add success. I1121 23:27:26.417742 3798194 cpu.go:282] Add success. I1121 23:27:26.441263 3798194 net.go:642] Add success. I1121 23:27:26.442780 3798194 net.go:778] primary dev: ETH0 I1121 23:27:26.442796 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:27:26.442810 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:27:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:27:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:27:29.450144 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:27:36.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:27:36.413257 3798194 memory.go:184] no items to output this cycle I1121 23:27:36.413410 3798194 cpu.go:275] no items to output this cycle E1121 23:27:46.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:27:46.412250 3798194 memory.go:184] no items to output this cycle I1121 23:27:46.412395 3798194 cpu.go:275] no items to output this cycle E1121 23:27:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:27:56.411946 3798194 memory.go:191] Add success. W1121 23:27:56.411974 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:27:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:27:56.411989 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:27:56.412357 3798194 cpu.go:282] Add success. I1121 23:27:56.442010 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:27:56.442773 3798194 net.go:642] Add success. I1121 23:27:56.447025 3798194 net.go:778] primary dev: ETH0 I1121 23:27:56.447042 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:27:56.447061 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:27:57.451014 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:27:57.451245 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:27:57.451258 3798194 disk_worker.go:723] disk space is not compliant W1121 23:27:57.451261 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:27:57.459967 3798194 disk_worker.go:509] system disk:vda1 I1121 23:27:57.460009 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:27:58.446015 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:27:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:27:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:27:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:27:59.471665 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:28:00.471138 3798194 disk_info.go:125] begin check local disk info of client I1121 23:28:00.475012 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:28:00.475021 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071280 0xc0000712c0] E1121 23:28:06.412090 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:28:06.412128 3798194 memory.go:184] no items to output this cycle I1121 23:28:06.412279 3798194 cpu.go:275] no items to output this cycle E1121 23:28:16.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:28:16.412263 3798194 memory.go:184] no items to output this cycle I1121 23:28:16.412412 3798194 cpu.go:275] no items to output this cycle E1121 23:28:26.411705 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:28:26.411739 3798194 memory.go:191] Add success. I1121 23:28:26.415934 3798194 cpu.go:282] Add success. I1121 23:28:26.441487 3798194 net.go:642] Add success. I1121 23:28:26.442946 3798194 net.go:778] primary dev: ETH0 I1121 23:28:26.442962 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:28:26.442975 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:28:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:28:29.450084 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:28:29.450118 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:28:36.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:28:36.411975 3798194 memory.go:184] no items to output this cycle I1121 23:28:36.412136 3798194 cpu.go:275] no items to output this cycle E1121 23:28:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:28:46.411981 3798194 memory.go:184] no items to output this cycle I1121 23:28:46.412139 3798194 cpu.go:275] no items to output this cycle E1121 23:28:56.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:28:56.413274 3798194 memory.go:191] Add success. W1121 23:28:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:28:56.413316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:28:56.413320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:28:56.417413 3798194 cpu.go:282] Add success. I1121 23:28:56.441813 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:28:56.442322 3798194 net.go:642] Add success. I1121 23:28:56.443769 3798194 net.go:778] primary dev: ETH0 I1121 23:28:56.443785 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:28:56.443797 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:28:57.451343 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:28:57.451506 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:28:57.451572 3798194 disk_worker.go:723] disk space is not compliant W1121 23:28:57.451576 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:28:57.461328 3798194 disk_worker.go:509] system disk:vda1 I1121 23:28:57.461375 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:28:58.446374 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:28:59.450406 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:28:59.450498 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:28:59.450524 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:28:59.471055 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:29:00.475103 3798194 disk_info.go:125] begin check local disk info of client I1121 23:29:00.479090 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:29:00.479099 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00042e040 0xc00042e080] E1121 23:29:06.413379 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:29:06.413401 3798194 memory.go:184] no items to output this cycle I1121 23:29:06.413542 3798194 cpu.go:275] no items to output this cycle E1121 23:29:16.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:29:16.412875 3798194 memory.go:184] no items to output this cycle I1121 23:29:16.413022 3798194 cpu.go:275] no items to output this cycle E1121 23:29:26.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:29:26.413274 3798194 memory.go:191] Add success. I1121 23:29:26.417300 3798194 cpu.go:282] Add success. I1121 23:29:26.441731 3798194 net.go:642] Add success. I1121 23:29:26.443154 3798194 net.go:778] primary dev: ETH0 I1121 23:29:26.443168 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:29:26.443183 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:29:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:29:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:29:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:29:36.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:29:36.412263 3798194 memory.go:184] no items to output this cycle I1121 23:29:36.412407 3798194 cpu.go:275] no items to output this cycle E1121 23:29:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:29:46.411978 3798194 memory.go:184] no items to output this cycle I1121 23:29:46.412163 3798194 cpu.go:275] no items to output this cycle E1121 23:29:56.413303 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:29:56.413342 3798194 memory.go:191] Add success. W1121 23:29:56.413370 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:29:56.413382 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:29:56.413385 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:29:56.420791 3798194 cpu.go:282] Add success. I1121 23:29:56.441189 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:29:56.441656 3798194 net.go:642] Add success. I1121 23:29:56.443237 3798194 net.go:778] primary dev: ETH0 I1121 23:29:56.443253 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:29:56.443266 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:29:57.450408 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:29:57.450606 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:29:57.450617 3798194 disk_worker.go:723] disk space is not compliant W1121 23:29:57.450620 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:29:57.458987 3798194 disk_worker.go:509] system disk:vda1 I1121 23:29:57.459041 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:29:58.446434 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:29:59.450069 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:29:59.450152 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:29:59.450178 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:29:59.471710 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:30:00.479557 3798194 disk_info.go:125] begin check local disk info of client I1121 23:30:00.483515 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:30:00.483524 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036a1c0 0xc00036a200] E1121 23:30:06.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:30:06.412248 3798194 memory.go:184] no items to output this cycle I1121 23:30:06.412392 3798194 cpu.go:275] no items to output this cycle E1121 23:30:16.412663 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:30:16.412688 3798194 memory.go:184] no items to output this cycle I1121 23:30:16.412833 3798194 cpu.go:275] no items to output this cycle I1121 23:30:17.576879 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:30:17.576888 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:30:26.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:30:26.412889 3798194 memory.go:191] Add success. I1121 23:30:26.417482 3798194 cpu.go:282] Add success. I1121 23:30:26.442031 3798194 net.go:642] Add success. I1121 23:30:26.443775 3798194 net.go:778] primary dev: ETH0 I1121 23:30:26.443790 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:30:26.443804 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:30:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:30:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:30:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:30:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:30:36.411976 3798194 memory.go:184] no items to output this cycle I1121 23:30:36.412134 3798194 cpu.go:275] no items to output this cycle E1121 23:30:46.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:30:46.413270 3798194 memory.go:184] no items to output this cycle I1121 23:30:46.413418 3798194 cpu.go:275] no items to output this cycle E1121 23:30:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:30:56.411950 3798194 memory.go:191] Add success. W1121 23:30:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:30:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:30:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:30:56.412349 3798194 cpu.go:282] Add success. I1121 23:30:56.444719 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:30:56.445220 3798194 net.go:642] Add success. I1121 23:30:56.449244 3798194 net.go:778] primary dev: ETH0 I1121 23:30:56.449260 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:30:56.449275 3798194 net.go:659] Add eni metrics success, count: 7 E1121 23:30:57.450673 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 23:30:57.450687 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:30:57.450840 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:30:57.450889 3798194 disk_worker.go:723] disk space is not compliant W1121 23:30:57.450893 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:30:57.459150 3798194 disk_worker.go:509] system disk:vda1 I1121 23:30:57.459190 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 23:30:58.448126 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 23:30:58.448145 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 23:30:59.448778 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 23:30:59.449853 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:30:59.449902 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:30:59.449923 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:30:59.471501 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:31:00.484202 3798194 disk_info.go:125] begin check local disk info of client I1121 23:31:00.488064 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:31:00.488074 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036a500 0xc00036a540] E1121 23:31:06.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:31:06.413262 3798194 memory.go:184] no items to output this cycle I1121 23:31:06.413403 3798194 cpu.go:275] no items to output this cycle E1121 23:31:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:31:16.411984 3798194 memory.go:184] no items to output this cycle I1121 23:31:16.412141 3798194 cpu.go:275] no items to output this cycle E1121 23:31:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:31:26.411958 3798194 memory.go:191] Add success. I1121 23:31:26.412295 3798194 cpu.go:282] Add success. I1121 23:31:26.441759 3798194 net.go:642] Add success. I1121 23:31:26.445632 3798194 net.go:778] primary dev: ETH0 I1121 23:31:26.445647 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:31:26.445662 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:31:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:31:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:31:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:31:36.412917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:31:36.412940 3798194 memory.go:184] no items to output this cycle I1121 23:31:36.413086 3798194 cpu.go:275] no items to output this cycle E1121 23:31:46.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:31:46.412256 3798194 memory.go:184] no items to output this cycle I1121 23:31:46.412404 3798194 cpu.go:275] no items to output this cycle E1121 23:31:56.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:31:56.411957 3798194 memory.go:191] Add success. W1121 23:31:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:31:56.412009 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:31:56.412012 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:31:56.412365 3798194 cpu.go:282] Add success. I1121 23:31:56.441836 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:31:56.442360 3798194 net.go:642] Add success. I1121 23:31:56.446080 3798194 net.go:778] primary dev: ETH0 I1121 23:31:56.446095 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:31:56.446107 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:31:57.451408 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:31:57.451577 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:31:57.451633 3798194 disk_worker.go:723] disk space is not compliant W1121 23:31:57.451636 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:31:57.460043 3798194 disk_worker.go:509] system disk:vda1 I1121 23:31:57.460083 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:31:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:31:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:31:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:31:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:31:59.471651 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:32:00.488179 3798194 disk_info.go:125] begin check local disk info of client I1121 23:32:00.492060 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:32:00.492068 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00048b9c0 0xc00048ba00] E1121 23:32:06.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:32:06.412261 3798194 memory.go:184] no items to output this cycle I1121 23:32:06.412417 3798194 cpu.go:275] no items to output this cycle E1121 23:32:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:32:16.411981 3798194 memory.go:184] no items to output this cycle I1121 23:32:16.412150 3798194 cpu.go:275] no items to output this cycle E1121 23:32:26.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:32:26.411958 3798194 memory.go:191] Add success. I1121 23:32:26.412320 3798194 cpu.go:282] Add success. I1121 23:32:26.441887 3798194 net.go:642] Add success. I1121 23:32:26.446200 3798194 net.go:778] primary dev: ETH0 I1121 23:32:26.446216 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:32:26.446230 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:32:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:32:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:32:29.450126 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:32:36.412079 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:32:36.412108 3798194 memory.go:184] no items to output this cycle I1121 23:32:36.412276 3798194 cpu.go:275] no items to output this cycle E1121 23:32:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:32:46.413254 3798194 memory.go:184] no items to output this cycle I1121 23:32:46.413401 3798194 cpu.go:275] no items to output this cycle E1121 23:32:56.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:32:56.412886 3798194 memory.go:191] Add success. W1121 23:32:56.412914 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:32:56.412926 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:32:56.412930 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:32:56.416937 3798194 cpu.go:282] Add success. I1121 23:32:56.441328 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:32:56.441834 3798194 net.go:642] Add success. I1121 23:32:56.443262 3798194 net.go:778] primary dev: ETH0 I1121 23:32:56.443279 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:32:56.443290 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:32:57.450557 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:32:57.450776 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:32:57.450787 3798194 disk_worker.go:723] disk space is not compliant W1121 23:32:57.450791 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:32:57.459073 3798194 disk_worker.go:509] system disk:vda1 I1121 23:32:57.459132 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:32:58.445994 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:32:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:32:59.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:32:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:32:59.471681 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:33:00.492146 3798194 disk_info.go:125] begin check local disk info of client I1121 23:33:00.496017 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:33:00.496028 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000402e40 0xc000402e80] E1121 23:33:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:33:06.413256 3798194 memory.go:184] no items to output this cycle I1121 23:33:06.413398 3798194 cpu.go:275] no items to output this cycle E1121 23:33:16.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:33:16.413268 3798194 memory.go:184] no items to output this cycle I1121 23:33:16.413605 3798194 cpu.go:275] no items to output this cycle I1121 23:33:17.577206 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:33:17.577215 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:33:26.411350 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:33:26.411386 3798194 memory.go:191] Add success. I1121 23:33:26.411748 3798194 cpu.go:282] Add success. I1121 23:33:26.441352 3798194 net.go:642] Add success. I1121 23:33:26.448738 3798194 net.go:778] primary dev: ETH0 I1121 23:33:26.448763 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:33:26.448780 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:33:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:33:29.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:33:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:33:36.413165 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:33:36.413187 3798194 memory.go:184] no items to output this cycle I1121 23:33:36.413331 3798194 cpu.go:275] no items to output this cycle E1121 23:33:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:33:46.411972 3798194 memory.go:184] no items to output this cycle I1121 23:33:46.412128 3798194 cpu.go:275] no items to output this cycle E1121 23:33:56.416232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:33:56.416263 3798194 memory.go:191] Add success. W1121 23:33:56.416290 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:33:56.416302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:33:56.416306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:33:56.441355 3798194 cpu.go:282] Add success. I1121 23:33:56.441531 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:33:56.442014 3798194 net.go:642] Add success. I1121 23:33:56.468229 3798194 net.go:778] primary dev: ETH0 I1121 23:33:56.468246 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:33:56.468261 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:33:57.445419 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:33:57.445668 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:33:57.445680 3798194 disk_worker.go:723] disk space is not compliant W1121 23:33:57.445683 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:33:57.447770 3798194 disk_worker.go:509] system disk:vda1 I1121 23:33:57.447809 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:33:58.446372 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:33:59.450005 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:33:59.450085 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:33:59.450121 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:33:59.471674 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:34:00.496898 3798194 disk_info.go:125] begin check local disk info of client I1121 23:34:00.500888 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:34:00.500898 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e9c40 0xc0002e9c80] E1121 23:34:06.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:34:06.412264 3798194 memory.go:184] no items to output this cycle I1121 23:34:06.412404 3798194 cpu.go:275] no items to output this cycle E1121 23:34:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:34:16.411985 3798194 memory.go:184] no items to output this cycle I1121 23:34:16.412146 3798194 cpu.go:275] no items to output this cycle E1121 23:34:26.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:34:26.413267 3798194 memory.go:191] Add success. I1121 23:34:26.417108 3798194 cpu.go:282] Add success. I1121 23:34:26.441920 3798194 net.go:642] Add success. I1121 23:34:26.443576 3798194 net.go:778] primary dev: ETH0 I1121 23:34:26.443592 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:34:26.443605 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:34:29.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:34:29.450460 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:34:29.450485 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:34:36.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:34:36.413252 3798194 memory.go:184] no items to output this cycle I1121 23:34:36.413412 3798194 cpu.go:275] no items to output this cycle E1121 23:34:46.412571 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:34:46.412597 3798194 memory.go:184] no items to output this cycle I1121 23:34:46.412899 3798194 cpu.go:275] no items to output this cycle E1121 23:34:56.413202 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:34:56.413250 3798194 memory.go:191] Add success. W1121 23:34:56.413285 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:34:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:34:56.413301 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:34:56.417574 3798194 cpu.go:282] Add success. I1121 23:34:56.441080 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:34:56.441659 3798194 net.go:642] Add success. I1121 23:34:56.443205 3798194 net.go:778] primary dev: ETH0 I1121 23:34:56.443228 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:34:56.443241 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:34:57.450848 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:34:57.451060 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:34:57.451072 3798194 disk_worker.go:723] disk space is not compliant W1121 23:34:57.451075 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:34:57.459951 3798194 disk_worker.go:509] system disk:vda1 I1121 23:34:57.460010 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:34:58.446464 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:34:59.450559 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:34:59.450639 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:34:59.450665 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:34:59.471129 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:35:00.501001 3798194 disk_info.go:125] begin check local disk info of client I1121 23:35:00.504864 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:35:00.504873 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e9dc0 0xc0002e9e00] E1121 23:35:06.413142 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:35:06.413170 3798194 memory.go:184] no items to output this cycle I1121 23:35:06.413314 3798194 cpu.go:275] no items to output this cycle E1121 23:35:16.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:35:16.412873 3798194 memory.go:184] no items to output this cycle I1121 23:35:16.413016 3798194 cpu.go:275] no items to output this cycle E1121 23:35:26.411235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:35:26.411267 3798194 memory.go:191] Add success. I1121 23:35:26.415180 3798194 cpu.go:282] Add success. I1121 23:35:26.441769 3798194 net.go:642] Add success. I1121 23:35:26.443246 3798194 net.go:778] primary dev: ETH0 I1121 23:35:26.443260 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:35:26.443274 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:35:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:35:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:35:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:35:36.412376 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:35:36.412399 3798194 memory.go:184] no items to output this cycle I1121 23:35:36.412539 3798194 cpu.go:275] no items to output this cycle E1121 23:35:46.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:35:46.411969 3798194 memory.go:184] no items to output this cycle I1121 23:35:46.412129 3798194 cpu.go:275] no items to output this cycle E1121 23:35:56.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:35:56.413269 3798194 memory.go:191] Add success. W1121 23:35:56.413302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:35:56.413313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:35:56.413317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:35:56.417180 3798194 cpu.go:282] Add success. I1121 23:35:56.444332 3798194 event_worker.go:152] Polling the log file for events... I1121 23:35:56.444696 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:35:56.445217 3798194 net.go:642] Add success. I1121 23:35:56.446571 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795273556, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1121 23:35:56.446702 3798194 net.go:778] primary dev: ETH0 I1121 23:35:56.446716 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:35:56.446729 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:35:56.456188 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1121 23:35:57.449211 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 23:35:57.449227 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:35:57.449407 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:35:57.449465 3798194 disk_worker.go:723] disk space is not compliant W1121 23:35:57.449468 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:35:57.452090 3798194 disk_worker.go:509] system disk:vda1 I1121 23:35:57.452148 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 23:35:58.448527 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 23:35:58.448543 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 23:35:59.448952 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 23:35:59.450037 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:35:59.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:35:59.450107 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:35:59.471644 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:36:00.505648 3798194 disk_info.go:125] begin check local disk info of client I1121 23:36:00.509581 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:36:00.509590 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00049d940 0xc00049d980] E1121 23:36:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:36:06.411982 3798194 memory.go:184] no items to output this cycle I1121 23:36:06.412146 3798194 cpu.go:275] no items to output this cycle E1121 23:36:16.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:36:16.413262 3798194 memory.go:184] no items to output this cycle I1121 23:36:16.413406 3798194 cpu.go:275] no items to output this cycle I1121 23:36:17.577371 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:36:17.577379 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:36:26.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:36:26.412269 3798194 memory.go:191] Add success. I1121 23:36:26.416503 3798194 cpu.go:282] Add success. I1121 23:36:26.442065 3798194 net.go:642] Add success. I1121 23:36:26.443633 3798194 net.go:778] primary dev: ETH0 I1121 23:36:26.443648 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:36:26.443661 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:36:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:36:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:36:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:36:36.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:36:36.411990 3798194 memory.go:184] no items to output this cycle I1121 23:36:36.412143 3798194 cpu.go:275] no items to output this cycle E1121 23:36:46.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:36:46.411985 3798194 memory.go:184] no items to output this cycle I1121 23:36:46.412144 3798194 cpu.go:275] no items to output this cycle E1121 23:36:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:36:56.412264 3798194 memory.go:191] Add success. W1121 23:36:56.412292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:36:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:36:56.412306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:36:56.416846 3798194 cpu.go:282] Add success. I1121 23:36:56.441228 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:36:56.441711 3798194 net.go:642] Add success. I1121 23:36:56.443504 3798194 net.go:778] primary dev: ETH0 I1121 23:36:56.443518 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:36:56.443530 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:36:57.450762 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:36:57.450931 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:36:57.450986 3798194 disk_worker.go:723] disk space is not compliant W1121 23:36:57.450990 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:36:57.459287 3798194 disk_worker.go:509] system disk:vda1 I1121 23:36:57.459329 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:36:58.446375 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:36:59.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:36:59.450459 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:36:59.450485 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:36:59.471019 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:37:00.510645 3798194 disk_info.go:125] begin check local disk info of client I1121 23:37:00.514536 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:37:00.514546 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e4e40 0xc0003e4e80] E1121 23:37:06.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:37:06.412884 3798194 memory.go:184] no items to output this cycle I1121 23:37:06.413024 3798194 cpu.go:275] no items to output this cycle E1121 23:37:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:37:16.411983 3798194 memory.go:184] no items to output this cycle I1121 23:37:16.412154 3798194 cpu.go:275] no items to output this cycle E1121 23:37:26.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:37:26.412278 3798194 memory.go:191] Add success. I1121 23:37:26.416451 3798194 cpu.go:282] Add success. I1121 23:37:26.442106 3798194 net.go:642] Add success. I1121 23:37:26.443559 3798194 net.go:778] primary dev: ETH0 I1121 23:37:26.443572 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:37:26.443585 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:37:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:37:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:37:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:37:36.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:37:36.413248 3798194 memory.go:184] no items to output this cycle I1121 23:37:36.413398 3798194 cpu.go:275] no items to output this cycle E1121 23:37:46.411789 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:37:46.411812 3798194 memory.go:184] no items to output this cycle I1121 23:37:46.411956 3798194 cpu.go:275] no items to output this cycle E1121 23:37:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:37:56.412274 3798194 memory.go:191] Add success. W1121 23:37:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:37:56.412318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:37:56.412321 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:37:56.416516 3798194 cpu.go:282] Add success. I1121 23:37:56.440933 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:37:56.441544 3798194 net.go:642] Add success. I1121 23:37:56.442933 3798194 net.go:778] primary dev: ETH0 I1121 23:37:56.442949 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:37:56.442962 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:37:57.449375 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:37:57.449625 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:37:57.449637 3798194 disk_worker.go:723] disk space is not compliant W1121 23:37:57.449640 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:37:57.451467 3798194 disk_worker.go:509] system disk:vda1 I1121 23:37:57.451508 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:37:58.446625 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:37:59.450581 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:37:59.450670 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:37:59.450696 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:37:59.471235 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:38:00.515231 3798194 disk_info.go:125] begin check local disk info of client I1121 23:38:00.519099 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:38:00.519125 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00027f440 0xc00027f480] E1121 23:38:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:38:06.411977 3798194 memory.go:184] no items to output this cycle I1121 23:38:06.412131 3798194 cpu.go:275] no items to output this cycle E1121 23:38:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:38:16.411972 3798194 memory.go:184] no items to output this cycle I1121 23:38:16.412125 3798194 cpu.go:275] no items to output this cycle E1121 23:38:26.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:38:26.411944 3798194 memory.go:191] Add success. I1121 23:38:26.412301 3798194 cpu.go:282] Add success. I1121 23:38:26.441911 3798194 net.go:642] Add success. I1121 23:38:26.445754 3798194 net.go:778] primary dev: ETH0 I1121 23:38:26.445769 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:38:26.445782 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:38:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:38:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:38:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:38:36.412098 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:38:36.412140 3798194 memory.go:184] no items to output this cycle I1121 23:38:36.412284 3798194 cpu.go:275] no items to output this cycle E1121 23:38:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:38:46.411982 3798194 memory.go:184] no items to output this cycle I1121 23:38:46.412139 3798194 cpu.go:275] no items to output this cycle E1121 23:38:56.411930 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:38:56.411968 3798194 memory.go:191] Add success. W1121 23:38:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:38:56.412009 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:38:56.412012 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:38:56.412396 3798194 cpu.go:282] Add success. I1121 23:38:56.441935 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:38:56.442530 3798194 net.go:642] Add success. I1121 23:38:56.446538 3798194 net.go:778] primary dev: ETH0 I1121 23:38:56.446553 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:38:56.446565 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:38:57.450912 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:38:57.451104 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:38:57.451145 3798194 disk_worker.go:723] disk space is not compliant W1121 23:38:57.451149 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:38:57.459842 3798194 disk_worker.go:509] system disk:vda1 I1121 23:38:57.459883 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:38:58.446013 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:38:59.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:38:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:38:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:38:59.471671 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:39:00.519236 3798194 disk_info.go:125] begin check local disk info of client I1121 23:39:00.523166 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:39:00.523175 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0d80 0xc0002e0dc0] E1121 23:39:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:39:06.411979 3798194 memory.go:184] no items to output this cycle I1121 23:39:06.412135 3798194 cpu.go:275] no items to output this cycle E1121 23:39:16.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:39:16.413262 3798194 memory.go:184] no items to output this cycle I1121 23:39:16.413407 3798194 cpu.go:275] no items to output this cycle I1121 23:39:17.577551 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:39:17.577559 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:39:26.412428 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:39:26.412461 3798194 memory.go:191] Add success. I1121 23:39:26.412776 3798194 cpu.go:282] Add success. I1121 23:39:26.441313 3798194 net.go:642] Add success. I1121 23:39:26.445569 3798194 net.go:778] primary dev: ETH0 I1121 23:39:26.445583 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:39:26.445595 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:39:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:39:29.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:39:29.450490 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:39:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:39:36.412257 3798194 memory.go:184] no items to output this cycle I1121 23:39:36.412398 3798194 cpu.go:275] no items to output this cycle E1121 23:39:46.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:39:46.412261 3798194 memory.go:184] no items to output this cycle I1121 23:39:46.412409 3798194 cpu.go:275] no items to output this cycle E1121 23:39:56.413196 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:39:56.413231 3798194 memory.go:191] Add success. W1121 23:39:56.413257 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:39:56.413277 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:39:56.413280 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:39:56.417560 3798194 cpu.go:282] Add success. I1121 23:39:56.440900 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:39:56.441489 3798194 net.go:642] Add success. I1121 23:39:56.442991 3798194 net.go:778] primary dev: ETH0 I1121 23:39:56.443007 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:39:56.443019 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:39:57.449073 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:39:57.449333 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:39:57.449345 3798194 disk_worker.go:723] disk space is not compliant W1121 23:39:57.449349 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:39:57.451152 3798194 disk_worker.go:509] system disk:vda1 I1121 23:39:57.451194 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:39:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:39:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:39:59.450128 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:39:59.450162 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:39:59.471643 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:40:00.523269 3798194 disk_info.go:125] begin check local disk info of client I1121 23:40:00.527238 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:40:00.527247 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00048e100 0xc00048e140] E1121 23:40:06.412976 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:40:06.413063 3798194 memory.go:184] no items to output this cycle I1121 23:40:06.413471 3798194 cpu.go:275] no items to output this cycle E1121 23:40:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:40:16.411977 3798194 memory.go:184] no items to output this cycle I1121 23:40:16.412149 3798194 cpu.go:275] no items to output this cycle E1121 23:40:26.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:40:26.413262 3798194 memory.go:191] Add success. I1121 23:40:26.417566 3798194 cpu.go:282] Add success. I1121 23:40:26.442069 3798194 net.go:642] Add success. I1121 23:40:26.443578 3798194 net.go:778] primary dev: ETH0 I1121 23:40:26.443594 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:40:26.443607 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:40:29.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:40:29.450466 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:40:29.450495 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:40:36.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:40:36.411983 3798194 memory.go:184] no items to output this cycle I1121 23:40:36.412138 3798194 cpu.go:275] no items to output this cycle E1121 23:40:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:40:46.413254 3798194 memory.go:184] no items to output this cycle I1121 23:40:46.413420 3798194 cpu.go:275] no items to output this cycle E1121 23:40:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:40:56.412268 3798194 memory.go:191] Add success. W1121 23:40:56.412294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:40:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:40:56.412309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:40:56.416317 3798194 cpu.go:282] Add success. I1121 23:40:56.444538 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:40:56.445108 3798194 net.go:642] Add success. I1121 23:40:56.446584 3798194 net.go:778] primary dev: ETH0 I1121 23:40:56.446599 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:40:56.446611 3798194 net.go:659] Add eni metrics success, count: 7 W1121 23:40:57.443889 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:40:57.443951 3798194 disk_worker.go:723] disk space is not compliant W1121 23:40:57.443954 3798194 disk_worker.go:743] disk inode is not compliant E1121 23:40:57.452196 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 23:40:57.452209 3798194 custom_config.go:106] query custom config with name: gpu I1121 23:40:57.452234 3798194 disk_worker.go:509] system disk:vda1 I1121 23:40:57.452273 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 23:40:58.448471 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 23:40:58.448485 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 23:40:59.448994 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 23:40:59.450068 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:40:59.450125 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:40:59.450146 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:40:59.471662 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:41:00.527348 3798194 disk_info.go:125] begin check local disk info of client I1121 23:41:00.531252 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:41:00.531261 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00035d4c0 0xc00035d500] E1121 23:41:06.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:41:06.411979 3798194 memory.go:184] no items to output this cycle I1121 23:41:06.412146 3798194 cpu.go:275] no items to output this cycle E1121 23:41:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:41:16.411973 3798194 memory.go:184] no items to output this cycle I1121 23:41:16.412127 3798194 cpu.go:275] no items to output this cycle E1121 23:41:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:41:26.411955 3798194 memory.go:191] Add success. I1121 23:41:26.412288 3798194 cpu.go:282] Add success. I1121 23:41:26.441744 3798194 net.go:642] Add success. I1121 23:41:26.445655 3798194 net.go:778] primary dev: ETH0 I1121 23:41:26.445670 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:41:26.445682 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:41:29.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:41:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:41:29.450501 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:41:36.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:41:36.413257 3798194 memory.go:184] no items to output this cycle I1121 23:41:36.413407 3798194 cpu.go:275] no items to output this cycle E1121 23:41:46.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:41:46.412266 3798194 memory.go:184] no items to output this cycle I1121 23:41:46.412415 3798194 cpu.go:275] no items to output this cycle E1121 23:41:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:41:56.411956 3798194 memory.go:191] Add success. W1121 23:41:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:41:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:41:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:41:56.412353 3798194 cpu.go:282] Add success. I1121 23:41:56.441824 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:41:56.442339 3798194 net.go:642] Add success. I1121 23:41:56.446975 3798194 net.go:778] primary dev: ETH0 I1121 23:41:56.446989 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:41:56.447002 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:41:57.451179 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:41:57.451341 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:41:57.451392 3798194 disk_worker.go:723] disk space is not compliant W1121 23:41:57.451395 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:41:57.459730 3798194 disk_worker.go:509] system disk:vda1 I1121 23:41:57.459774 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:41:58.446194 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:41:59.450414 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:41:59.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:41:59.450508 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:41:59.471027 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:42:00.531362 3798194 disk_info.go:125] begin check local disk info of client I1121 23:42:00.535213 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:42:00.535222 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b5380 0xc0003b53c0] E1121 23:42:06.413171 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:42:06.413197 3798194 memory.go:184] no items to output this cycle I1121 23:42:06.413343 3798194 cpu.go:275] no items to output this cycle E1121 23:42:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:42:16.413255 3798194 memory.go:184] no items to output this cycle I1121 23:42:16.413398 3798194 cpu.go:275] no items to output this cycle I1121 23:42:17.577724 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:42:17.577733 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:42:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:42:26.413264 3798194 memory.go:191] Add success. I1121 23:42:26.417734 3798194 cpu.go:282] Add success. I1121 23:42:26.441639 3798194 net.go:642] Add success. I1121 23:42:26.443090 3798194 net.go:778] primary dev: ETH0 I1121 23:42:26.443105 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:42:26.443126 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:42:29.450517 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:42:29.450586 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:42:29.450610 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:42:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:42:36.412252 3798194 memory.go:184] no items to output this cycle I1121 23:42:36.412395 3798194 cpu.go:275] no items to output this cycle E1121 23:42:46.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:42:46.413258 3798194 memory.go:184] no items to output this cycle I1121 23:42:46.413400 3798194 cpu.go:275] no items to output this cycle E1121 23:42:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:42:56.412270 3798194 memory.go:191] Add success. W1121 23:42:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:42:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:42:56.412311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:42:56.416471 3798194 cpu.go:282] Add success. I1121 23:42:56.440922 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:42:56.441844 3798194 net.go:642] Add success. I1121 23:42:56.444027 3798194 net.go:778] primary dev: ETH0 I1121 23:42:56.444041 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:42:56.444053 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:42:57.445415 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:42:57.445665 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:42:57.445677 3798194 disk_worker.go:723] disk space is not compliant W1121 23:42:57.445680 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:42:57.447773 3798194 disk_worker.go:509] system disk:vda1 I1121 23:42:57.447815 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:42:58.445986 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:42:59.450500 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:42:59.450567 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:42:59.450591 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:42:59.471116 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:43:00.535329 3798194 disk_info.go:125] begin check local disk info of client I1121 23:43:00.539211 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:43:00.539220 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047ee00 0xc00047ee40] E1121 23:43:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:43:06.411986 3798194 memory.go:184] no items to output this cycle I1121 23:43:06.412141 3798194 cpu.go:275] no items to output this cycle E1121 23:43:16.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:43:16.412256 3798194 memory.go:184] no items to output this cycle I1121 23:43:16.412413 3798194 cpu.go:275] no items to output this cycle E1121 23:43:26.412543 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:43:26.412575 3798194 memory.go:191] Add success. I1121 23:43:26.417761 3798194 cpu.go:282] Add success. I1121 23:43:26.441404 3798194 net.go:642] Add success. I1121 23:43:26.442888 3798194 net.go:778] primary dev: ETH0 I1121 23:43:26.442905 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:43:26.442917 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:43:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:43:29.450086 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:43:29.450122 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:43:36.413256 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:43:36.413278 3798194 memory.go:184] no items to output this cycle I1121 23:43:36.413419 3798194 cpu.go:275] no items to output this cycle E1121 23:43:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:43:46.411977 3798194 memory.go:184] no items to output this cycle I1121 23:43:46.412133 3798194 cpu.go:275] no items to output this cycle E1121 23:43:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:43:56.412269 3798194 memory.go:191] Add success. W1121 23:43:56.412295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:43:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:43:56.412310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:43:56.416544 3798194 cpu.go:282] Add success. I1121 23:43:56.441222 3798194 net.go:642] Add success. I1121 23:43:56.442956 3798194 net.go:778] primary dev: ETH0 I1121 23:43:56.442971 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:43:56.442993 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:43:56.441554 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:43:57.449268 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:43:57.449456 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:43:57.449510 3798194 disk_worker.go:723] disk space is not compliant W1121 23:43:57.449513 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:43:57.451372 3798194 disk_worker.go:509] system disk:vda1 I1121 23:43:57.451412 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:43:58.445632 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:43:59.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:43:59.450461 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:43:59.450487 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:43:59.471003 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:44:00.539315 3798194 disk_info.go:125] begin check local disk info of client I1121 23:44:00.543249 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:44:00.543259 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f8900 0xc0003f8940] E1121 23:44:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:44:06.411975 3798194 memory.go:184] no items to output this cycle I1121 23:44:06.412249 3798194 cpu.go:275] no items to output this cycle E1121 23:44:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:44:16.411977 3798194 memory.go:184] no items to output this cycle I1121 23:44:16.412135 3798194 cpu.go:275] no items to output this cycle E1121 23:44:26.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:44:26.411948 3798194 memory.go:191] Add success. I1121 23:44:26.412321 3798194 cpu.go:282] Add success. I1121 23:44:26.441897 3798194 net.go:642] Add success. I1121 23:44:26.445697 3798194 net.go:778] primary dev: ETH0 I1121 23:44:26.445712 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:44:26.445726 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:44:29.450257 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:44:29.450337 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:44:29.450373 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:44:36.413146 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:44:36.413170 3798194 memory.go:184] no items to output this cycle I1121 23:44:36.413313 3798194 cpu.go:275] no items to output this cycle E1121 23:44:46.412707 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:44:46.412728 3798194 memory.go:184] no items to output this cycle I1121 23:44:46.412874 3798194 cpu.go:275] no items to output this cycle E1121 23:44:56.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:44:56.412890 3798194 memory.go:191] Add success. W1121 23:44:56.412916 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:44:56.412928 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:44:56.412932 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:44:56.417297 3798194 cpu.go:282] Add success. I1121 23:44:56.441675 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:44:56.442180 3798194 net.go:642] Add success. I1121 23:44:56.443707 3798194 net.go:778] primary dev: ETH0 I1121 23:44:56.443722 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:44:56.443734 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:44:57.450654 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:44:57.450851 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:44:57.450862 3798194 disk_worker.go:723] disk space is not compliant W1121 23:44:57.450865 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:44:57.459215 3798194 disk_worker.go:509] system disk:vda1 I1121 23:44:57.459258 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:44:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:44:59.450098 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:44:59.450189 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:44:59.450215 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:44:59.470693 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:45:00.543363 3798194 disk_info.go:125] begin check local disk info of client I1121 23:45:00.547194 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:45:00.547204 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000372f80 0xc000372fc0] E1121 23:45:06.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:45:06.412876 3798194 memory.go:184] no items to output this cycle I1121 23:45:06.413018 3798194 cpu.go:275] no items to output this cycle E1121 23:45:16.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:45:16.411969 3798194 memory.go:184] no items to output this cycle I1121 23:45:16.412126 3798194 cpu.go:275] no items to output this cycle I1121 23:45:17.578730 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:45:17.578739 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:45:26.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:45:26.413260 3798194 memory.go:191] Add success. I1121 23:45:26.417471 3798194 cpu.go:282] Add success. I1121 23:45:26.442025 3798194 net.go:642] Add success. I1121 23:45:26.443505 3798194 net.go:778] primary dev: ETH0 I1121 23:45:26.443520 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:45:26.443533 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:45:29.450000 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:45:29.450077 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:45:29.450101 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:45:36.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:45:36.411973 3798194 memory.go:184] no items to output this cycle I1121 23:45:36.412333 3798194 cpu.go:275] no items to output this cycle E1121 23:45:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:45:46.411978 3798194 memory.go:184] no items to output this cycle I1121 23:45:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 23:45:56.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:45:56.413260 3798194 memory.go:191] Add success. W1121 23:45:56.413286 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:45:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:45:56.413300 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:45:56.417928 3798194 cpu.go:282] Add success. I1121 23:45:56.442543 3798194 event_worker.go:152] Polling the log file for events... I1121 23:45:56.442767 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:45:56.443276 3798194 net.go:642] Add success. I1121 23:45:56.446215 3798194 net.go:778] primary dev: ETH0 I1121 23:45:56.446231 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:45:56.446244 3798194 net.go:659] Add eni metrics success, count: 7 E1121 23:45:57.446484 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 23:45:57.446499 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:45:57.446746 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:45:57.446758 3798194 disk_worker.go:723] disk space is not compliant W1121 23:45:57.446761 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:45:57.449621 3798194 disk_worker.go:509] system disk:vda1 I1121 23:45:57.449662 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 23:45:58.448474 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 23:45:58.448488 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 23:45:59.448272 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 23:45:59.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:45:59.450453 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:45:59.450472 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:45:59.471005 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:46:00.547306 3798194 disk_info.go:125] begin check local disk info of client I1121 23:46:00.551175 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:46:00.551185 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047f9c0 0xc00047fa00] E1121 23:46:06.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:46:06.413273 3798194 memory.go:184] no items to output this cycle I1121 23:46:06.413420 3798194 cpu.go:275] no items to output this cycle E1121 23:46:16.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:46:16.412870 3798194 memory.go:184] no items to output this cycle I1121 23:46:16.413015 3798194 cpu.go:275] no items to output this cycle E1121 23:46:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:46:26.411954 3798194 memory.go:191] Add success. I1121 23:46:26.412303 3798194 cpu.go:282] Add success. I1121 23:46:26.441763 3798194 net.go:642] Add success. I1121 23:46:26.446005 3798194 net.go:778] primary dev: ETH0 I1121 23:46:26.446021 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:46:26.446034 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:46:29.449664 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:46:29.449741 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:46:29.449769 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:46:36.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:46:36.413264 3798194 memory.go:184] no items to output this cycle I1121 23:46:36.413405 3798194 cpu.go:275] no items to output this cycle E1121 23:46:46.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:46:46.411986 3798194 memory.go:184] no items to output this cycle I1121 23:46:46.412142 3798194 cpu.go:275] no items to output this cycle E1121 23:46:56.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:46:56.411961 3798194 memory.go:191] Add success. W1121 23:46:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:46:56.412009 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:46:56.412012 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:46:56.412370 3798194 cpu.go:282] Add success. I1121 23:46:56.441819 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:46:56.442414 3798194 net.go:642] Add success. I1121 23:46:56.446814 3798194 net.go:778] primary dev: ETH0 I1121 23:46:56.446830 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:46:56.446850 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:46:57.450791 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:46:57.450998 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:46:57.451009 3798194 disk_worker.go:723] disk space is not compliant W1121 23:46:57.451012 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:46:57.459718 3798194 disk_worker.go:509] system disk:vda1 I1121 23:46:57.459760 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:46:58.446015 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:46:59.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:46:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:46:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:46:59.471703 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:47:00.551289 3798194 disk_info.go:125] begin check local disk info of client I1121 23:47:00.555227 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:47:00.555237 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071000 0xc000071040] E1121 23:47:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:47:06.412260 3798194 memory.go:184] no items to output this cycle I1121 23:47:06.412402 3798194 cpu.go:275] no items to output this cycle E1121 23:47:16.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:47:16.412877 3798194 memory.go:184] no items to output this cycle I1121 23:47:16.413028 3798194 cpu.go:275] no items to output this cycle E1121 23:47:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:47:26.413267 3798194 memory.go:191] Add success. I1121 23:47:26.417854 3798194 cpu.go:282] Add success. I1121 23:47:26.441280 3798194 net.go:642] Add success. I1121 23:47:26.442779 3798194 net.go:778] primary dev: ETH0 I1121 23:47:26.442794 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:47:26.442807 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:47:29.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:47:29.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:47:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:47:36.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:47:36.411993 3798194 memory.go:184] no items to output this cycle I1121 23:47:36.412146 3798194 cpu.go:275] no items to output this cycle E1121 23:47:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:47:46.411982 3798194 memory.go:184] no items to output this cycle I1121 23:47:46.412138 3798194 cpu.go:275] no items to output this cycle E1121 23:47:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:47:56.411944 3798194 memory.go:191] Add success. W1121 23:47:56.411981 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:47:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:47:56.411996 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:47:56.412351 3798194 cpu.go:282] Add success. I1121 23:47:56.441845 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:47:56.442410 3798194 net.go:642] Add success. I1121 23:47:56.446750 3798194 net.go:778] primary dev: ETH0 I1121 23:47:56.446768 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:47:56.446780 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:47:57.450581 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:47:57.450798 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:47:57.450809 3798194 disk_worker.go:723] disk space is not compliant W1121 23:47:57.450812 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:47:57.459264 3798194 disk_worker.go:509] system disk:vda1 I1121 23:47:57.459305 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:47:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:47:59.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:47:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:47:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:47:59.471645 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:48:00.555332 3798194 disk_info.go:125] begin check local disk info of client I1121 23:48:00.559204 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:48:00.559217 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000426000 0xc000426040] E1121 23:48:06.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:48:06.413271 3798194 memory.go:184] no items to output this cycle I1121 23:48:06.413414 3798194 cpu.go:275] no items to output this cycle E1121 23:48:16.413372 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:48:16.413397 3798194 memory.go:184] no items to output this cycle I1121 23:48:16.413673 3798194 cpu.go:275] no items to output this cycle I1121 23:48:17.578970 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:48:17.578978 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:48:26.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:48:26.413271 3798194 memory.go:191] Add success. I1121 23:48:26.417616 3798194 cpu.go:282] Add success. I1121 23:48:26.442049 3798194 net.go:642] Add success. I1121 23:48:26.443508 3798194 net.go:778] primary dev: ETH0 I1121 23:48:26.443523 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:48:26.443538 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:48:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:48:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:48:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:48:36.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:48:36.411980 3798194 memory.go:184] no items to output this cycle I1121 23:48:36.412136 3798194 cpu.go:275] no items to output this cycle E1121 23:48:46.413324 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:48:46.413350 3798194 memory.go:184] no items to output this cycle I1121 23:48:46.413500 3798194 cpu.go:275] no items to output this cycle E1121 23:48:56.411434 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:48:56.411470 3798194 memory.go:191] Add success. W1121 23:48:56.411502 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:48:56.411514 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:48:56.411519 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:48:56.415878 3798194 cpu.go:282] Add success. I1121 23:48:56.441323 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:48:56.441803 3798194 net.go:642] Add success. I1121 23:48:56.443260 3798194 net.go:778] primary dev: ETH0 I1121 23:48:56.443284 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:48:56.443297 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:48:57.451371 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:48:57.451585 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:48:57.451596 3798194 disk_worker.go:723] disk space is not compliant W1121 23:48:57.451599 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:48:57.460082 3798194 disk_worker.go:509] system disk:vda1 I1121 23:48:57.460140 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:48:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:48:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:48:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:48:59.450132 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:48:59.471667 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:49:00.559321 3798194 disk_info.go:125] begin check local disk info of client I1121 23:49:00.563157 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:49:00.563182 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003680c0 0xc000368100] E1121 23:49:06.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:49:06.412875 3798194 memory.go:184] no items to output this cycle I1121 23:49:06.413016 3798194 cpu.go:275] no items to output this cycle E1121 23:49:16.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:49:16.411969 3798194 memory.go:184] no items to output this cycle I1121 23:49:16.412132 3798194 cpu.go:275] no items to output this cycle E1121 23:49:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:49:26.413270 3798194 memory.go:191] Add success. I1121 23:49:26.417479 3798194 cpu.go:282] Add success. I1121 23:49:26.442011 3798194 net.go:642] Add success. I1121 23:49:26.443544 3798194 net.go:778] primary dev: ETH0 I1121 23:49:26.443559 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:49:26.443572 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:49:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:49:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:49:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:49:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:49:36.411980 3798194 memory.go:184] no items to output this cycle I1121 23:49:36.412269 3798194 cpu.go:275] no items to output this cycle E1121 23:49:46.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:49:46.412262 3798194 memory.go:184] no items to output this cycle I1121 23:49:46.412409 3798194 cpu.go:275] no items to output this cycle E1121 23:49:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:49:56.413268 3798194 memory.go:191] Add success. W1121 23:49:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:49:56.413309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:49:56.413312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:49:56.417943 3798194 cpu.go:282] Add success. I1121 23:49:56.441319 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:49:56.441878 3798194 net.go:642] Add success. I1121 23:49:56.444012 3798194 net.go:778] primary dev: ETH0 I1121 23:49:56.444027 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:49:56.444040 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:49:57.449243 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:49:57.449482 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:49:57.449493 3798194 disk_worker.go:723] disk space is not compliant W1121 23:49:57.449497 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:49:57.451312 3798194 disk_worker.go:509] system disk:vda1 I1121 23:49:57.451354 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:49:58.446374 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:49:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:49:59.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:49:59.450494 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:49:59.471019 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:50:00.563285 3798194 disk_info.go:125] begin check local disk info of client I1121 23:50:00.567204 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:50:00.567213 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000450500 0xc000450540] E1121 23:50:06.413355 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:50:06.413377 3798194 memory.go:184] no items to output this cycle I1121 23:50:06.413523 3798194 cpu.go:275] no items to output this cycle E1121 23:50:16.412882 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:50:16.412903 3798194 memory.go:184] no items to output this cycle I1121 23:50:16.413053 3798194 cpu.go:275] no items to output this cycle E1121 23:50:26.412601 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:50:26.412637 3798194 memory.go:191] Add success. I1121 23:50:26.417260 3798194 cpu.go:282] Add success. I1121 23:50:26.441731 3798194 net.go:642] Add success. I1121 23:50:26.443310 3798194 net.go:778] primary dev: ETH0 I1121 23:50:26.443324 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:50:26.443340 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:50:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:50:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:50:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:50:36.412846 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:50:36.412866 3798194 memory.go:184] no items to output this cycle I1121 23:50:36.413010 3798194 cpu.go:275] no items to output this cycle E1121 23:50:46.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:50:46.412263 3798194 memory.go:184] no items to output this cycle I1121 23:50:46.412408 3798194 cpu.go:275] no items to output this cycle E1121 23:50:56.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:50:56.413277 3798194 memory.go:191] Add success. W1121 23:50:56.413317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:50:56.413331 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:50:56.413334 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:50:56.417444 3798194 cpu.go:282] Add success. I1121 23:50:56.444685 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:50:56.445165 3798194 net.go:642] Add success. I1121 23:50:56.447141 3798194 net.go:778] primary dev: ETH0 I1121 23:50:56.447157 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:50:56.447176 3798194 net.go:659] Add eni metrics success, count: 7 E1121 23:50:57.451200 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 23:50:57.451213 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:50:57.451430 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:50:57.451440 3798194 disk_worker.go:723] disk space is not compliant W1121 23:50:57.451443 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:50:57.459752 3798194 disk_worker.go:509] system disk:vda1 I1121 23:50:57.459794 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 23:50:58.448665 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 23:50:58.448680 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 23:50:59.448271 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 23:50:59.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:50:59.450452 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:50:59.450471 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:50:59.470995 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:51:00.567315 3798194 disk_info.go:125] begin check local disk info of client I1121 23:51:00.571212 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:51:00.571224 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e7c0 0xc00030e800] E1121 23:51:06.413280 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:51:06.413304 3798194 memory.go:184] no items to output this cycle I1121 23:51:06.413447 3798194 cpu.go:275] no items to output this cycle E1121 23:51:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:51:16.411975 3798194 memory.go:184] no items to output this cycle I1121 23:51:16.412139 3798194 cpu.go:275] no items to output this cycle I1121 23:51:17.579146 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:51:17.579156 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:51:26.411348 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:51:26.411382 3798194 memory.go:191] Add success. I1121 23:51:26.411718 3798194 cpu.go:282] Add success. I1121 23:51:26.441301 3798194 net.go:642] Add success. I1121 23:51:26.445325 3798194 net.go:778] primary dev: ETH0 I1121 23:51:26.445343 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:51:26.445357 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:51:29.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:51:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:51:29.450494 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:51:36.413159 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:51:36.413188 3798194 memory.go:184] no items to output this cycle I1121 23:51:36.413332 3798194 cpu.go:275] no items to output this cycle E1121 23:51:46.413142 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:51:46.413164 3798194 memory.go:184] no items to output this cycle I1121 23:51:46.413318 3798194 cpu.go:275] no items to output this cycle E1121 23:51:56.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:51:56.412890 3798194 memory.go:191] Add success. W1121 23:51:56.412924 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:51:56.412936 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:51:56.412939 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:51:56.417309 3798194 cpu.go:282] Add success. I1121 23:51:56.441736 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:51:56.442275 3798194 net.go:642] Add success. I1121 23:51:56.443770 3798194 net.go:778] primary dev: ETH0 I1121 23:51:56.443786 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:51:56.443799 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:51:57.451102 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:51:57.451297 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:51:57.451354 3798194 disk_worker.go:723] disk space is not compliant W1121 23:51:57.451357 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:51:57.459736 3798194 disk_worker.go:509] system disk:vda1 I1121 23:51:57.459781 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:51:58.446546 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:51:59.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:51:59.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:51:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:51:59.471640 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:52:00.571319 3798194 disk_info.go:125] begin check local disk info of client I1121 23:52:00.575213 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:52:00.575223 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000406100 0xc000406140] E1121 23:52:06.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:52:06.412875 3798194 memory.go:184] no items to output this cycle I1121 23:52:06.413016 3798194 cpu.go:275] no items to output this cycle E1121 23:52:16.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:52:16.413263 3798194 memory.go:184] no items to output this cycle I1121 23:52:16.413414 3798194 cpu.go:275] no items to output this cycle E1121 23:52:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:52:26.411959 3798194 memory.go:191] Add success. I1121 23:52:26.412328 3798194 cpu.go:282] Add success. I1121 23:52:26.441919 3798194 net.go:642] Add success. I1121 23:52:26.445910 3798194 net.go:778] primary dev: ETH0 I1121 23:52:26.445926 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:52:26.445939 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:52:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:52:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:52:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:52:36.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:52:36.412261 3798194 memory.go:184] no items to output this cycle I1121 23:52:36.412408 3798194 cpu.go:275] no items to output this cycle E1121 23:52:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:52:46.413255 3798194 memory.go:184] no items to output this cycle I1121 23:52:46.413408 3798194 cpu.go:275] no items to output this cycle E1121 23:52:56.411933 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:52:56.411969 3798194 memory.go:191] Add success. W1121 23:52:56.412001 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:52:56.412013 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:52:56.412016 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:52:56.412375 3798194 cpu.go:282] Add success. I1121 23:52:56.441922 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:52:56.442539 3798194 net.go:642] Add success. I1121 23:52:56.447080 3798194 net.go:778] primary dev: ETH0 I1121 23:52:56.447095 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:52:56.447108 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:52:57.454169 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:52:57.454338 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:52:57.454392 3798194 disk_worker.go:723] disk space is not compliant W1121 23:52:57.454395 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:52:57.470176 3798194 disk_worker.go:509] system disk:vda1 I1121 23:52:57.470220 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:52:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:52:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:52:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:52:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:52:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:53:00.575324 3798194 disk_info.go:125] begin check local disk info of client I1121 23:53:00.579180 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:53:00.579188 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00039e340 0xc00039e380] E1121 23:53:06.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:53:06.411969 3798194 memory.go:184] no items to output this cycle I1121 23:53:06.412118 3798194 cpu.go:275] no items to output this cycle E1121 23:53:16.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:53:16.412263 3798194 memory.go:184] no items to output this cycle I1121 23:53:16.412410 3798194 cpu.go:275] no items to output this cycle E1121 23:53:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:53:26.411954 3798194 memory.go:191] Add success. I1121 23:53:26.412324 3798194 cpu.go:282] Add success. I1121 23:53:26.441980 3798194 net.go:642] Add success. I1121 23:53:26.446092 3798194 net.go:778] primary dev: ETH0 I1121 23:53:26.446131 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:53:26.446150 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:53:29.449991 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:53:29.450062 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:53:29.450087 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:53:36.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:53:36.413270 3798194 memory.go:184] no items to output this cycle I1121 23:53:36.413622 3798194 cpu.go:275] no items to output this cycle E1121 23:53:46.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:53:46.412264 3798194 memory.go:184] no items to output this cycle I1121 23:53:46.412414 3798194 cpu.go:275] no items to output this cycle E1121 23:53:56.412617 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:53:56.412649 3798194 memory.go:191] Add success. W1121 23:53:56.412677 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:53:56.412688 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:53:56.412693 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:53:56.416756 3798194 cpu.go:282] Add success. I1121 23:53:56.441182 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:53:56.441681 3798194 net.go:642] Add success. I1121 23:53:56.443137 3798194 net.go:778] primary dev: ETH0 I1121 23:53:56.443152 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:53:56.443165 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:53:57.450771 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:53:57.450969 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:53:57.450980 3798194 disk_worker.go:723] disk space is not compliant W1121 23:53:57.450983 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:53:57.459816 3798194 disk_worker.go:509] system disk:vda1 I1121 23:53:57.459857 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:53:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:53:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:53:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:53:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:53:59.471652 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:54:00.579246 3798194 disk_info.go:125] begin check local disk info of client I1121 23:54:00.583218 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:54:00.583226 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047f3c0 0xc00047f400] E1121 23:54:06.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:54:06.413258 3798194 memory.go:184] no items to output this cycle I1121 23:54:06.413406 3798194 cpu.go:275] no items to output this cycle E1121 23:54:16.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:54:16.413260 3798194 memory.go:184] no items to output this cycle I1121 23:54:16.413408 3798194 cpu.go:275] no items to output this cycle I1121 23:54:17.579325 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:54:17.579334 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:54:26.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:54:26.412272 3798194 memory.go:191] Add success. I1121 23:54:26.416465 3798194 cpu.go:282] Add success. I1121 23:54:26.442006 3798194 net.go:642] Add success. I1121 23:54:26.443492 3798194 net.go:778] primary dev: ETH0 I1121 23:54:26.443507 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:54:26.443522 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:54:29.450193 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:54:29.450277 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:54:29.450306 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:54:36.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:54:36.412261 3798194 memory.go:184] no items to output this cycle I1121 23:54:36.412399 3798194 cpu.go:275] no items to output this cycle E1121 23:54:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:54:46.411977 3798194 memory.go:184] no items to output this cycle I1121 23:54:46.412136 3798194 cpu.go:275] no items to output this cycle E1121 23:54:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:54:56.411956 3798194 memory.go:191] Add success. W1121 23:54:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:54:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:54:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:54:56.412352 3798194 cpu.go:282] Add success. I1121 23:54:56.441810 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:54:56.442533 3798194 net.go:642] Add success. I1121 23:54:56.446692 3798194 net.go:778] primary dev: ETH0 I1121 23:54:56.446711 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:54:56.446730 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:54:57.450714 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:54:57.450876 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:54:57.450933 3798194 disk_worker.go:723] disk space is not compliant W1121 23:54:57.450936 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:54:57.459814 3798194 disk_worker.go:509] system disk:vda1 I1121 23:54:57.459854 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:54:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:54:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:54:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:54:59.450132 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:54:59.471629 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:55:00.583330 3798194 disk_info.go:125] begin check local disk info of client I1121 23:55:00.587257 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:55:00.587267 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000070f00 0xc000070f40] E1121 23:55:06.412763 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:55:06.412785 3798194 memory.go:184] no items to output this cycle I1121 23:55:06.412935 3798194 cpu.go:275] no items to output this cycle E1121 23:55:16.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:55:16.411987 3798194 memory.go:184] no items to output this cycle I1121 23:55:16.412146 3798194 cpu.go:275] no items to output this cycle E1121 23:55:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:55:26.411956 3798194 memory.go:191] Add success. I1121 23:55:26.412296 3798194 cpu.go:282] Add success. I1121 23:55:26.441748 3798194 net.go:642] Add success. I1121 23:55:26.445989 3798194 net.go:778] primary dev: ETH0 I1121 23:55:26.446004 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:55:26.446016 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:55:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:55:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:55:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:55:36.413160 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:55:36.413182 3798194 memory.go:184] no items to output this cycle I1121 23:55:36.413339 3798194 cpu.go:275] no items to output this cycle E1121 23:55:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:55:46.412258 3798194 memory.go:184] no items to output this cycle I1121 23:55:46.412403 3798194 cpu.go:275] no items to output this cycle E1121 23:55:56.411827 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:55:56.411860 3798194 memory.go:191] Add success. W1121 23:55:56.411888 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:55:56.411899 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:55:56.411903 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:55:56.416013 3798194 cpu.go:282] Add success. I1121 23:55:56.444002 3798194 event_worker.go:152] Polling the log file for events... I1121 23:55:56.444234 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:55:56.444698 3798194 net.go:642] Add success. I1121 23:55:56.446286 3798194 net.go:778] primary dev: ETH0 I1121 23:55:56.446300 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:55:56.446311 3798194 net.go:659] Add eni metrics success, count: 7 E1121 23:55:57.450590 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1121 23:55:57.450603 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:55:57.450810 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:55:57.450820 3798194 disk_worker.go:723] disk space is not compliant W1121 23:55:57.450823 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:55:57.459133 3798194 disk_worker.go:509] system disk:vda1 I1121 23:55:57.459175 3798194 disk_worker.go:434] add disk info successfully, len:34 E1121 23:55:58.447781 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1121 23:55:58.447795 3798194 custom_config.go:106] query custom config with name: huawei_npu E1121 23:55:59.448599 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1121 23:55:59.449672 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:55:59.449724 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:55:59.449745 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:55:59.471185 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:56:00.587360 3798194 disk_info.go:125] begin check local disk info of client I1121 23:56:00.591271 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:56:00.591281 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000410a40 0xc000410a80] E1121 23:56:06.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:56:06.413259 3798194 memory.go:184] no items to output this cycle I1121 23:56:06.413538 3798194 cpu.go:275] no items to output this cycle E1121 23:56:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:56:16.413263 3798194 memory.go:184] no items to output this cycle I1121 23:56:16.413410 3798194 cpu.go:275] no items to output this cycle E1121 23:56:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:56:26.411957 3798194 memory.go:191] Add success. I1121 23:56:26.412336 3798194 cpu.go:282] Add success. I1121 23:56:26.441917 3798194 net.go:642] Add success. I1121 23:56:26.445734 3798194 net.go:778] primary dev: ETH0 I1121 23:56:26.445750 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:56:26.445764 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:56:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:56:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:56:29.450151 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:56:36.412959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:56:36.412980 3798194 memory.go:184] no items to output this cycle I1121 23:56:36.413140 3798194 cpu.go:275] no items to output this cycle E1121 23:56:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:56:46.411981 3798194 memory.go:184] no items to output this cycle I1121 23:56:46.412151 3798194 cpu.go:275] no items to output this cycle E1121 23:56:56.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:56:56.412888 3798194 memory.go:191] Add success. W1121 23:56:56.412923 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:56:56.412936 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:56:56.412940 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:56:56.417622 3798194 cpu.go:282] Add success. I1121 23:56:56.440957 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:56:56.441546 3798194 net.go:642] Add success. I1121 23:56:56.443096 3798194 net.go:778] primary dev: ETH0 I1121 23:56:56.443123 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:56:56.443138 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:56:57.453523 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:56:57.453734 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:56:57.453745 3798194 disk_worker.go:723] disk space is not compliant W1121 23:56:57.453748 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:56:57.464991 3798194 disk_worker.go:509] system disk:vda1 I1121 23:56:57.465035 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:56:58.445996 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:56:59.450095 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:56:59.450182 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:56:59.450207 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:56:59.470669 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:57:00.591323 3798194 disk_info.go:125] begin check local disk info of client I1121 23:57:00.595238 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:57:00.595248 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000369000 0xc000369040] E1121 23:57:06.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:57:06.413251 3798194 memory.go:184] no items to output this cycle I1121 23:57:06.413393 3798194 cpu.go:275] no items to output this cycle E1121 23:57:16.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:57:16.413266 3798194 memory.go:184] no items to output this cycle I1121 23:57:16.413421 3798194 cpu.go:275] no items to output this cycle I1121 23:57:17.579502 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1121 23:57:17.579510 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1121 23:57:26.411452 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:57:26.411485 3798194 memory.go:191] Add success. I1121 23:57:26.411823 3798194 cpu.go:282] Add success. I1121 23:57:26.441427 3798194 net.go:642] Add success. I1121 23:57:26.445680 3798194 net.go:778] primary dev: ETH0 I1121 23:57:26.445695 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:57:26.445708 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:57:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:57:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:57:29.450512 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:57:36.413175 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:57:36.413199 3798194 memory.go:184] no items to output this cycle I1121 23:57:36.413341 3798194 cpu.go:275] no items to output this cycle E1121 23:57:46.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:57:46.413258 3798194 memory.go:184] no items to output this cycle I1121 23:57:46.413402 3798194 cpu.go:275] no items to output this cycle E1121 23:57:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:57:56.412267 3798194 memory.go:191] Add success. W1121 23:57:56.412298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:57:56.412310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:57:56.412313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:57:56.416764 3798194 cpu.go:282] Add success. I1121 23:57:56.441186 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:57:56.441655 3798194 net.go:642] Add success. I1121 23:57:56.443185 3798194 net.go:778] primary dev: ETH0 I1121 23:57:56.443200 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:57:56.443214 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:57:57.450688 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:57:57.450907 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:57:57.450919 3798194 disk_worker.go:723] disk space is not compliant W1121 23:57:57.450922 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:57:57.459273 3798194 disk_worker.go:509] system disk:vda1 I1121 23:57:57.459315 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:57:58.445877 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:57:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:57:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:57:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:57:59.471691 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:58:00.595351 3798194 disk_info.go:125] begin check local disk info of client I1121 23:58:00.599291 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:58:00.599300 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d3180 0xc0003d31c0] E1121 23:58:06.412224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:58:06.412247 3798194 memory.go:184] no items to output this cycle I1121 23:58:06.412400 3798194 cpu.go:275] no items to output this cycle E1121 23:58:16.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:58:16.412884 3798194 memory.go:184] no items to output this cycle I1121 23:58:16.413026 3798194 cpu.go:275] no items to output this cycle E1121 23:58:26.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:58:26.413271 3798194 memory.go:191] Add success. I1121 23:58:26.417873 3798194 cpu.go:282] Add success. I1121 23:58:26.441434 3798194 net.go:642] Add success. I1121 23:58:26.443018 3798194 net.go:778] primary dev: ETH0 I1121 23:58:26.443034 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:58:26.443046 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:58:29.449670 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:58:29.449745 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:58:29.449772 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:58:36.411835 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:58:36.411856 3798194 memory.go:184] no items to output this cycle I1121 23:58:36.412001 3798194 cpu.go:275] no items to output this cycle E1121 23:58:46.412262 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:58:46.412287 3798194 memory.go:184] no items to output this cycle I1121 23:58:46.412440 3798194 cpu.go:275] no items to output this cycle E1121 23:58:56.412334 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:58:56.412379 3798194 memory.go:191] Add success. W1121 23:58:56.412429 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:58:56.412447 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:58:56.412452 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:58:56.420931 3798194 cpu.go:282] Add success. I1121 23:58:56.441695 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:58:56.442537 3798194 net.go:642] Add success. I1121 23:58:56.444037 3798194 net.go:778] primary dev: ETH0 I1121 23:58:56.444054 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:58:56.444068 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:58:57.451365 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:58:57.451579 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:58:57.451590 3798194 disk_worker.go:723] disk space is not compliant W1121 23:58:57.451593 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:58:57.462068 3798194 disk_worker.go:509] system disk:vda1 I1121 23:58:57.462128 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:58:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:58:59.450409 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:58:59.450495 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:58:59.450521 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:58:59.471057 3798194 disk_local_worker.go:436] Get disk info: [] I1121 23:59:00.599405 3798194 disk_info.go:125] begin check local disk info of client I1121 23:59:00.603481 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1121 23:59:00.603490 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000466c00 0xc000466c40] E1121 23:59:06.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:59:06.411990 3798194 memory.go:184] no items to output this cycle I1121 23:59:06.412143 3798194 cpu.go:275] no items to output this cycle E1121 23:59:16.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:59:16.413269 3798194 memory.go:184] no items to output this cycle I1121 23:59:16.413422 3798194 cpu.go:275] no items to output this cycle E1121 23:59:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:59:26.411951 3798194 memory.go:191] Add success. I1121 23:59:26.412306 3798194 cpu.go:282] Add success. I1121 23:59:26.441927 3798194 net.go:642] Add success. I1121 23:59:26.445941 3798194 net.go:778] primary dev: ETH0 I1121 23:59:26.445958 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:59:26.445970 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:59:29.449669 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:59:29.449751 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:59:29.449776 3798194 gpu_kunlun.go:227] Add success, len:1 E1121 23:59:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:59:36.413255 3798194 memory.go:184] no items to output this cycle I1121 23:59:36.413402 3798194 cpu.go:275] no items to output this cycle E1121 23:59:46.411943 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:59:46.411967 3798194 memory.go:184] no items to output this cycle I1121 23:59:46.412125 3798194 cpu.go:275] no items to output this cycle E1121 23:59:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1121 23:59:56.412264 3798194 memory.go:191] Add success. W1121 23:59:56.412290 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1121 23:59:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1121 23:59:56.412305 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1121 23:59:56.416610 3798194 cpu.go:282] Add success. I1121 23:59:56.441107 3798194 controller.go:85] Send host metrics successfully, count is 5 I1121 23:59:56.441593 3798194 net.go:642] Add success. I1121 23:59:56.443280 3798194 net.go:778] primary dev: ETH0 I1121 23:59:56.443294 3798194 net.go:810] Send network stats successfully!,count is 6 I1121 23:59:56.443306 3798194 net.go:659] Add eni metrics success, count: 7 I1121 23:59:57.451475 3798194 custom_config.go:106] query custom config with name: gpu W1121 23:59:57.451678 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1121 23:59:57.451688 3798194 disk_worker.go:723] disk space is not compliant W1121 23:59:57.451691 3798194 disk_worker.go:743] disk inode is not compliant I1121 23:59:57.459980 3798194 disk_worker.go:509] system disk:vda1 I1121 23:59:57.460023 3798194 disk_worker.go:434] add disk info successfully, len:34 I1121 23:59:58.446381 3798194 custom_config.go:106] query custom config with name: huawei_npu I1121 23:59:59.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1121 23:59:59.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1121 23:59:59.450507 3798194 gpu_kunlun.go:227] Add success, len:1 I1121 23:59:59.471050 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:00:00.603590 3798194 disk_info.go:125] begin check local disk info of client I1122 00:00:00.607432 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:00:00.607441 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e8880 0xc0003e88c0] E1122 00:00:06.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:00:06.412267 3798194 memory.go:184] no items to output this cycle I1122 00:00:06.412411 3798194 cpu.go:275] no items to output this cycle E1122 00:00:16.413060 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:00:16.413087 3798194 memory.go:184] no items to output this cycle I1122 00:00:16.413252 3798194 cpu.go:275] no items to output this cycle I1122 00:00:17.579625 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:00:17.579633 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:00:26.411306 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:00:26.411338 3798194 memory.go:191] Add success. I1122 00:00:26.411697 3798194 cpu.go:282] Add success. I1122 00:00:26.441278 3798194 net.go:642] Add success. I1122 00:00:26.445398 3798194 net.go:778] primary dev: ETH0 I1122 00:00:26.445413 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:00:26.445426 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:00:29.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:00:29.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:00:29.450495 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:00:36.412354 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:00:36.412378 3798194 memory.go:184] no items to output this cycle I1122 00:00:36.412529 3798194 cpu.go:275] no items to output this cycle E1122 00:00:46.411943 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:00:46.411966 3798194 memory.go:184] no items to output this cycle I1122 00:00:46.412107 3798194 cpu.go:275] no items to output this cycle E1122 00:00:56.412079 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:00:56.412138 3798194 memory.go:191] Add success. W1122 00:00:56.412184 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:00:56.412199 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:00:56.412204 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:00:56.416610 3798194 cpu.go:282] Add success. I1122 00:00:56.443869 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:00:56.444457 3798194 net.go:642] Add success. I1122 00:00:56.446209 3798194 net.go:778] primary dev: ETH0 I1122 00:00:56.446224 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:00:56.446237 3798194 net.go:659] Add eni metrics success, count: 7 E1122 00:00:57.450649 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 00:00:57.450664 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:00:57.450858 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:00:57.450870 3798194 disk_worker.go:723] disk space is not compliant W1122 00:00:57.450873 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:00:57.459278 3798194 disk_worker.go:509] system disk:vda1 I1122 00:00:57.459318 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 00:00:58.448046 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 00:00:58.448060 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 00:00:59.448616 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 00:00:59.449691 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:00:59.449740 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:00:59.449758 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:00:59.471186 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:01:00.607541 3798194 disk_info.go:125] begin check local disk info of client I1122 00:01:00.611393 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:01:00.611405 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037c4c0 0xc00037c500] E1122 00:01:06.412872 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:01:06.412893 3798194 memory.go:184] no items to output this cycle I1122 00:01:06.413038 3798194 cpu.go:275] no items to output this cycle E1122 00:01:16.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:01:16.413271 3798194 memory.go:184] no items to output this cycle I1122 00:01:16.413420 3798194 cpu.go:275] no items to output this cycle E1122 00:01:26.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:01:26.412882 3798194 memory.go:191] Add success. I1122 00:01:26.417095 3798194 cpu.go:282] Add success. I1122 00:01:26.441541 3798194 net.go:642] Add success. I1122 00:01:26.443149 3798194 net.go:778] primary dev: ETH0 I1122 00:01:26.443165 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:01:26.443178 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:01:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:01:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:01:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:01:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:01:36.413264 3798194 memory.go:184] no items to output this cycle I1122 00:01:36.413604 3798194 cpu.go:275] no items to output this cycle E1122 00:01:46.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:01:46.412887 3798194 memory.go:184] no items to output this cycle I1122 00:01:46.413033 3798194 cpu.go:275] no items to output this cycle E1122 00:01:56.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:01:56.412264 3798194 memory.go:191] Add success. W1122 00:01:56.412294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:01:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:01:56.412309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:01:56.416828 3798194 cpu.go:282] Add success. I1122 00:01:56.441239 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:01:56.441726 3798194 net.go:642] Add success. I1122 00:01:56.443339 3798194 net.go:778] primary dev: ETH0 I1122 00:01:56.443353 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:01:56.443367 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:01:57.450983 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:01:57.451175 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:01:57.451234 3798194 disk_worker.go:723] disk space is not compliant W1122 00:01:57.451237 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:01:57.459987 3798194 disk_worker.go:509] system disk:vda1 I1122 00:01:57.460030 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:01:58.446423 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:01:59.450444 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:01:59.450532 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:01:59.450558 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:01:59.471123 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:02:00.611509 3798194 disk_info.go:125] begin check local disk info of client I1122 00:02:00.615350 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:02:00.615358 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000439300 0xc000439340] E1122 00:02:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:02:06.411985 3798194 memory.go:184] no items to output this cycle I1122 00:02:06.412154 3798194 cpu.go:275] no items to output this cycle E1122 00:02:16.412991 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:02:16.413012 3798194 memory.go:184] no items to output this cycle I1122 00:02:16.413173 3798194 cpu.go:275] no items to output this cycle E1122 00:02:26.411932 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:02:26.411964 3798194 memory.go:191] Add success. I1122 00:02:26.412318 3798194 cpu.go:282] Add success. I1122 00:02:26.441942 3798194 net.go:642] Add success. I1122 00:02:26.446339 3798194 net.go:778] primary dev: ETH0 I1122 00:02:26.446355 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:02:26.446371 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:02:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:02:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:02:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:02:36.412763 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:02:36.412786 3798194 memory.go:184] no items to output this cycle I1122 00:02:36.412938 3798194 cpu.go:275] no items to output this cycle E1122 00:02:46.412754 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:02:46.412778 3798194 memory.go:184] no items to output this cycle I1122 00:02:46.412922 3798194 cpu.go:275] no items to output this cycle E1122 00:02:56.412382 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:02:56.412418 3798194 memory.go:191] Add success. W1122 00:02:56.412452 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:02:56.412465 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:02:56.412470 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:02:56.417063 3798194 cpu.go:282] Add success. I1122 00:02:56.441418 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:02:56.441892 3798194 net.go:642] Add success. I1122 00:02:56.443560 3798194 net.go:778] primary dev: ETH0 I1122 00:02:56.443575 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:02:56.443588 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:02:57.450913 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:02:57.451354 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:02:57.451370 3798194 disk_worker.go:723] disk space is not compliant W1122 00:02:57.451373 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:02:57.460348 3798194 disk_worker.go:509] system disk:vda1 I1122 00:02:57.460388 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:02:58.446018 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:02:59.450126 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:02:59.450202 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:02:59.450226 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:02:59.470696 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:03:00.615466 3798194 disk_info.go:125] begin check local disk info of client I1122 00:03:00.619372 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:03:00.619382 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004679c0 0xc000467a00] E1122 00:03:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:03:06.411975 3798194 memory.go:184] no items to output this cycle I1122 00:03:06.412131 3798194 cpu.go:275] no items to output this cycle E1122 00:03:16.412351 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:03:16.412376 3798194 memory.go:184] no items to output this cycle I1122 00:03:16.412522 3798194 cpu.go:275] no items to output this cycle I1122 00:03:17.579795 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:03:17.579803 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:03:26.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:03:26.412886 3798194 memory.go:191] Add success. I1122 00:03:26.417236 3798194 cpu.go:282] Add success. I1122 00:03:26.441676 3798194 net.go:642] Add success. I1122 00:03:26.443251 3798194 net.go:778] primary dev: ETH0 I1122 00:03:26.443265 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:03:26.443277 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:03:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:03:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:03:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:03:36.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:03:36.411987 3798194 memory.go:184] no items to output this cycle I1122 00:03:36.412136 3798194 cpu.go:275] no items to output this cycle E1122 00:03:46.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:03:46.412260 3798194 memory.go:184] no items to output this cycle I1122 00:03:46.412406 3798194 cpu.go:275] no items to output this cycle E1122 00:03:56.412248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:03:56.412280 3798194 memory.go:191] Add success. W1122 00:03:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:03:56.412325 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:03:56.412328 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:03:56.416632 3798194 cpu.go:282] Add success. I1122 00:03:56.440961 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:03:56.441566 3798194 net.go:642] Add success. I1122 00:03:56.443261 3798194 net.go:778] primary dev: ETH0 I1122 00:03:56.443278 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:03:56.443290 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:03:57.450661 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:03:57.450885 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:03:57.450896 3798194 disk_worker.go:723] disk space is not compliant W1122 00:03:57.450900 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:03:57.459300 3798194 disk_worker.go:509] system disk:vda1 I1122 00:03:57.459342 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:03:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:03:59.450122 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:03:59.450193 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:03:59.450221 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:03:59.470713 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:04:00.620137 3798194 disk_info.go:125] begin check local disk info of client I1122 00:04:00.623976 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:04:00.623985 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d2c00 0xc0003d2c40] E1122 00:04:06.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:04:06.411985 3798194 memory.go:184] no items to output this cycle I1122 00:04:06.412143 3798194 cpu.go:275] no items to output this cycle E1122 00:04:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:04:16.411984 3798194 memory.go:184] no items to output this cycle I1122 00:04:16.412149 3798194 cpu.go:275] no items to output this cycle E1122 00:04:26.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:04:26.411957 3798194 memory.go:191] Add success. I1122 00:04:26.412749 3798194 cpu.go:282] Add success. I1122 00:04:26.441303 3798194 net.go:642] Add success. I1122 00:04:26.445403 3798194 net.go:778] primary dev: ETH0 I1122 00:04:26.445418 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:04:26.445432 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:04:29.450206 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:04:29.450291 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:04:29.450324 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:04:36.413177 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:04:36.413197 3798194 memory.go:184] no items to output this cycle I1122 00:04:36.413340 3798194 cpu.go:275] no items to output this cycle E1122 00:04:46.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:04:46.413252 3798194 memory.go:184] no items to output this cycle I1122 00:04:46.413393 3798194 cpu.go:275] no items to output this cycle E1122 00:04:56.412868 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:04:56.412902 3798194 memory.go:191] Add success. W1122 00:04:56.412937 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:04:56.412948 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:04:56.412951 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:04:56.417278 3798194 cpu.go:282] Add success. I1122 00:04:56.441665 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:04:56.442181 3798194 net.go:642] Add success. I1122 00:04:56.443772 3798194 net.go:778] primary dev: ETH0 I1122 00:04:56.443786 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:04:56.443799 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:04:57.451351 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:04:57.451564 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:04:57.451576 3798194 disk_worker.go:723] disk space is not compliant W1122 00:04:57.451579 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:04:57.459930 3798194 disk_worker.go:509] system disk:vda1 I1122 00:04:57.459975 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:04:58.446023 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:04:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:04:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:04:59.450133 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:04:59.471651 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:05:00.624087 3798194 disk_info.go:125] begin check local disk info of client I1122 00:05:00.627936 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:05:00.627945 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00027e880 0xc00027e8c0] E1122 00:05:06.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:05:06.412882 3798194 memory.go:184] no items to output this cycle I1122 00:05:06.413028 3798194 cpu.go:275] no items to output this cycle E1122 00:05:16.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:05:16.412257 3798194 memory.go:184] no items to output this cycle I1122 00:05:16.412402 3798194 cpu.go:275] no items to output this cycle E1122 00:05:26.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:05:26.411952 3798194 memory.go:191] Add success. I1122 00:05:26.412317 3798194 cpu.go:282] Add success. I1122 00:05:26.441748 3798194 net.go:642] Add success. I1122 00:05:26.445841 3798194 net.go:778] primary dev: ETH0 I1122 00:05:26.445856 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:05:26.445869 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:05:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:05:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:05:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:05:36.412834 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:05:36.412857 3798194 memory.go:184] no items to output this cycle I1122 00:05:36.413000 3798194 cpu.go:275] no items to output this cycle E1122 00:05:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:05:46.411975 3798194 memory.go:184] no items to output this cycle I1122 00:05:46.412136 3798194 cpu.go:275] no items to output this cycle E1122 00:05:56.412846 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:05:56.412885 3798194 memory.go:191] Add success. W1122 00:05:56.412915 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:05:56.412926 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:05:56.412929 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:05:56.417218 3798194 cpu.go:282] Add success. I1122 00:05:56.441398 3798194 event_worker.go:152] Polling the log file for events... I1122 00:05:56.444623 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:05:56.445078 3798194 net.go:642] Add success. I1122 00:05:56.446422 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795275356, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 00:05:56.446532 3798194 net.go:778] primary dev: ETH0 I1122 00:05:56.446547 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:05:56.446560 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:05:56.456187 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 00:05:57.445943 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 00:05:57.445961 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:05:57.446235 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:05:57.446332 3798194 disk_worker.go:723] disk space is not compliant W1122 00:05:57.446337 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:05:57.450363 3798194 disk_worker.go:509] system disk:vda1 I1122 00:05:57.450422 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 00:05:58.448102 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 00:05:58.448137 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 00:05:59.448624 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 00:05:59.449658 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:05:59.449711 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:05:59.449737 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:05:59.471205 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:06:00.628049 3798194 disk_info.go:125] begin check local disk info of client I1122 00:06:00.631926 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:06:00.631936 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e05c0 0xc0002e0600] E1122 00:06:06.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:06:06.413247 3798194 memory.go:184] no items to output this cycle I1122 00:06:06.413389 3798194 cpu.go:275] no items to output this cycle E1122 00:06:16.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:06:16.412879 3798194 memory.go:184] no items to output this cycle I1122 00:06:16.413032 3798194 cpu.go:275] no items to output this cycle I1122 00:06:17.580213 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:06:17.580221 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:06:26.411397 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:06:26.411431 3798194 memory.go:191] Add success. I1122 00:06:26.411780 3798194 cpu.go:282] Add success. I1122 00:06:26.441305 3798194 net.go:642] Add success. I1122 00:06:26.445325 3798194 net.go:778] primary dev: ETH0 I1122 00:06:26.445340 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:06:26.445352 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:06:29.450115 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:06:29.450211 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:06:29.450243 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:06:36.412428 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:06:36.412450 3798194 memory.go:184] no items to output this cycle I1122 00:06:36.412593 3798194 cpu.go:275] no items to output this cycle E1122 00:06:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:06:46.411979 3798194 memory.go:184] no items to output this cycle I1122 00:06:46.412140 3798194 cpu.go:275] no items to output this cycle E1122 00:06:56.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:06:56.411959 3798194 memory.go:191] Add success. W1122 00:06:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:06:56.412003 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:06:56.412007 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:06:56.412347 3798194 cpu.go:282] Add success. I1122 00:06:56.441794 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:06:56.442390 3798194 net.go:642] Add success. I1122 00:06:56.446886 3798194 net.go:778] primary dev: ETH0 I1122 00:06:56.446901 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:06:56.446914 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:06:57.450749 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:06:57.450993 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:06:57.451005 3798194 disk_worker.go:723] disk space is not compliant W1122 00:06:57.451008 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:06:57.461515 3798194 disk_worker.go:509] system disk:vda1 I1122 00:06:57.461578 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:06:58.446015 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:06:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:06:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:06:59.450127 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:06:59.471647 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:07:00.632039 3798194 disk_info.go:125] begin check local disk info of client I1122 00:07:00.635883 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:07:00.635891 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000449500 0xc000449940] E1122 00:07:06.412173 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:07:06.412195 3798194 memory.go:184] no items to output this cycle I1122 00:07:06.412348 3798194 cpu.go:275] no items to output this cycle E1122 00:07:16.413018 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:07:16.413042 3798194 memory.go:184] no items to output this cycle I1122 00:07:16.413218 3798194 cpu.go:275] no items to output this cycle E1122 00:07:26.413001 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:07:26.413037 3798194 memory.go:191] Add success. I1122 00:07:26.417284 3798194 cpu.go:282] Add success. I1122 00:07:26.441750 3798194 net.go:642] Add success. I1122 00:07:26.443404 3798194 net.go:778] primary dev: ETH0 I1122 00:07:26.443419 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:07:26.443432 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:07:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:07:29.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:07:29.450511 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:07:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:07:36.413260 3798194 memory.go:184] no items to output this cycle I1122 00:07:36.413404 3798194 cpu.go:275] no items to output this cycle E1122 00:07:46.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:07:46.413251 3798194 memory.go:184] no items to output this cycle I1122 00:07:46.413402 3798194 cpu.go:275] no items to output this cycle E1122 00:07:56.412842 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:07:56.412873 3798194 memory.go:191] Add success. W1122 00:07:56.412903 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:07:56.412915 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:07:56.412918 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:07:56.417588 3798194 cpu.go:282] Add success. I1122 00:07:56.440913 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:07:56.441445 3798194 net.go:642] Add success. I1122 00:07:56.443232 3798194 net.go:778] primary dev: ETH0 I1122 00:07:56.443248 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:07:56.443260 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:07:57.446273 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:07:57.446561 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:07:57.446573 3798194 disk_worker.go:723] disk space is not compliant W1122 00:07:57.446577 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:07:57.451600 3798194 disk_worker.go:509] system disk:vda1 I1122 00:07:57.451641 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:07:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:07:59.450479 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:07:59.450557 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:07:59.450582 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:07:59.471107 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:08:00.635992 3798194 disk_info.go:125] begin check local disk info of client I1122 00:08:00.639870 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:08:00.639879 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a2bc0 0xc0003a2c00] E1122 00:08:06.413304 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:08:06.413333 3798194 memory.go:184] no items to output this cycle I1122 00:08:06.413543 3798194 cpu.go:275] no items to output this cycle E1122 00:08:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:08:16.411979 3798194 memory.go:184] no items to output this cycle I1122 00:08:16.412137 3798194 cpu.go:275] no items to output this cycle E1122 00:08:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:08:26.411959 3798194 memory.go:191] Add success. I1122 00:08:26.412336 3798194 cpu.go:282] Add success. I1122 00:08:26.442051 3798194 net.go:642] Add success. I1122 00:08:26.446072 3798194 net.go:778] primary dev: ETH0 I1122 00:08:26.446087 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:08:26.446100 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:08:29.450466 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:08:29.450547 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:08:29.450578 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:08:36.412967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:08:36.412988 3798194 memory.go:184] no items to output this cycle I1122 00:08:36.413149 3798194 cpu.go:275] no items to output this cycle E1122 00:08:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:08:46.411978 3798194 memory.go:184] no items to output this cycle I1122 00:08:46.412139 3798194 cpu.go:275] no items to output this cycle E1122 00:08:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:08:56.412265 3798194 memory.go:191] Add success. W1122 00:08:56.412293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:08:56.412304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:08:56.412307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:08:56.416463 3798194 cpu.go:282] Add success. I1122 00:08:56.441840 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:08:56.442449 3798194 net.go:642] Add success. I1122 00:08:56.444004 3798194 net.go:778] primary dev: ETH0 I1122 00:08:56.444018 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:08:56.444030 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:08:57.450677 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:08:57.450835 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:08:57.450888 3798194 disk_worker.go:723] disk space is not compliant W1122 00:08:57.450891 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:08:57.459279 3798194 disk_worker.go:509] system disk:vda1 I1122 00:08:57.459318 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:08:58.446009 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:08:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:08:59.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:08:59.450147 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:08:59.470677 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:09:00.639988 3798194 disk_info.go:125] begin check local disk info of client I1122 00:09:00.643839 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:09:00.643848 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000327400 0xc000327440] E1122 00:09:06.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:09:06.412873 3798194 memory.go:184] no items to output this cycle I1122 00:09:06.413020 3798194 cpu.go:275] no items to output this cycle E1122 00:09:16.413253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:09:16.413274 3798194 memory.go:184] no items to output this cycle I1122 00:09:16.413421 3798194 cpu.go:275] no items to output this cycle I1122 00:09:17.580394 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:09:17.580403 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:09:26.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:09:26.413273 3798194 memory.go:191] Add success. I1122 00:09:26.417619 3798194 cpu.go:282] Add success. I1122 00:09:26.442181 3798194 net.go:642] Add success. I1122 00:09:26.443675 3798194 net.go:778] primary dev: ETH0 I1122 00:09:26.443689 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:09:26.443704 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:09:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:09:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:09:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:09:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:09:36.413262 3798194 memory.go:184] no items to output this cycle I1122 00:09:36.413403 3798194 cpu.go:275] no items to output this cycle E1122 00:09:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:09:46.411983 3798194 memory.go:184] no items to output this cycle I1122 00:09:46.412352 3798194 cpu.go:275] no items to output this cycle E1122 00:09:56.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:09:56.412894 3798194 memory.go:191] Add success. W1122 00:09:56.412925 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:09:56.412936 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:09:56.412940 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:09:56.417674 3798194 cpu.go:282] Add success. I1122 00:09:56.440963 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:09:56.441554 3798194 net.go:642] Add success. I1122 00:09:56.443124 3798194 net.go:778] primary dev: ETH0 I1122 00:09:56.443139 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:09:56.443151 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:09:57.450973 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:09:57.451209 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:09:57.451220 3798194 disk_worker.go:723] disk space is not compliant W1122 00:09:57.451223 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:09:57.465611 3798194 disk_worker.go:509] system disk:vda1 I1122 00:09:57.465657 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:09:58.445986 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:09:59.450500 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:09:59.450588 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:09:59.450615 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:09:59.471137 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:10:00.643957 3798194 disk_info.go:125] begin check local disk info of client I1122 00:10:00.647836 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:10:00.647846 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002f8d80 0xc0002f8dc0] E1122 00:10:06.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:10:06.411969 3798194 memory.go:184] no items to output this cycle I1122 00:10:06.412123 3798194 cpu.go:275] no items to output this cycle E1122 00:10:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:10:16.411977 3798194 memory.go:184] no items to output this cycle I1122 00:10:16.412143 3798194 cpu.go:275] no items to output this cycle E1122 00:10:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:10:26.412266 3798194 memory.go:191] Add success. I1122 00:10:26.416788 3798194 cpu.go:282] Add success. I1122 00:10:26.441292 3798194 net.go:642] Add success. I1122 00:10:26.442905 3798194 net.go:778] primary dev: ETH0 I1122 00:10:26.442920 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:10:26.442931 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:10:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:10:29.450125 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:10:29.450153 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:10:36.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:10:36.412871 3798194 memory.go:184] no items to output this cycle I1122 00:10:36.413026 3798194 cpu.go:275] no items to output this cycle E1122 00:10:46.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:10:46.413251 3798194 memory.go:184] no items to output this cycle I1122 00:10:46.413408 3798194 cpu.go:275] no items to output this cycle E1122 00:10:56.411909 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:10:56.411940 3798194 memory.go:191] Add success. W1122 00:10:56.411968 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:10:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:10:56.411982 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:10:56.412349 3798194 cpu.go:282] Add success. I1122 00:10:56.444752 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:10:56.445277 3798194 net.go:642] Add success. I1122 00:10:56.449435 3798194 net.go:778] primary dev: ETH0 I1122 00:10:56.449451 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:10:56.449466 3798194 net.go:659] Add eni metrics success, count: 7 E1122 00:10:57.450539 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 00:10:57.450553 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:10:57.450701 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:10:57.450751 3798194 disk_worker.go:723] disk space is not compliant W1122 00:10:57.450754 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:10:57.459866 3798194 disk_worker.go:509] system disk:vda1 I1122 00:10:57.459906 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 00:10:58.448085 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 00:10:58.448127 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 00:10:59.448650 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 00:10:59.449727 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:10:59.449781 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:10:59.449803 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:10:59.471400 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:11:00.648146 3798194 disk_info.go:125] begin check local disk info of client I1122 00:11:00.652067 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:11:00.652075 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000433600 0xc000433640] E1122 00:11:06.413254 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:11:06.413276 3798194 memory.go:184] no items to output this cycle I1122 00:11:06.413420 3798194 cpu.go:275] no items to output this cycle E1122 00:11:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:11:16.411973 3798194 memory.go:184] no items to output this cycle I1122 00:11:16.412143 3798194 cpu.go:275] no items to output this cycle E1122 00:11:26.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:11:26.413263 3798194 memory.go:191] Add success. I1122 00:11:26.417996 3798194 cpu.go:282] Add success. I1122 00:11:26.441513 3798194 net.go:642] Add success. I1122 00:11:26.443251 3798194 net.go:778] primary dev: ETH0 I1122 00:11:26.443265 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:11:26.443279 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:11:29.450416 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:11:29.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:11:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:11:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:11:36.413264 3798194 memory.go:184] no items to output this cycle I1122 00:11:36.413410 3798194 cpu.go:275] no items to output this cycle E1122 00:11:46.412991 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:11:46.413013 3798194 memory.go:184] no items to output this cycle I1122 00:11:46.413169 3798194 cpu.go:275] no items to output this cycle E1122 00:11:56.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:11:56.412261 3798194 memory.go:191] Add success. W1122 00:11:56.412295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:11:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:11:56.412310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:11:56.416613 3798194 cpu.go:282] Add success. I1122 00:11:56.440942 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:11:56.441511 3798194 net.go:642] Add success. I1122 00:11:56.442985 3798194 net.go:778] primary dev: ETH0 I1122 00:11:56.442999 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:11:56.443010 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:11:57.445495 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:11:57.445767 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:11:57.445779 3798194 disk_worker.go:723] disk space is not compliant W1122 00:11:57.445782 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:11:57.447861 3798194 disk_worker.go:509] system disk:vda1 I1122 00:11:57.447903 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:11:58.445990 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:11:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:11:59.450128 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:11:59.450156 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:11:59.471656 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:12:00.652171 3798194 disk_info.go:125] begin check local disk info of client I1122 00:12:00.656187 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:12:00.656197 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c6c40 0xc0003c6c80] E1122 00:12:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:12:06.412261 3798194 memory.go:184] no items to output this cycle I1122 00:12:06.412403 3798194 cpu.go:275] no items to output this cycle E1122 00:12:16.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:12:16.412255 3798194 memory.go:184] no items to output this cycle I1122 00:12:16.412402 3798194 cpu.go:275] no items to output this cycle I1122 00:12:17.581225 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:12:17.581233 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:12:26.413098 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:12:26.413153 3798194 memory.go:191] Add success. I1122 00:12:26.417878 3798194 cpu.go:282] Add success. I1122 00:12:26.442225 3798194 net.go:642] Add success. I1122 00:12:26.443830 3798194 net.go:778] primary dev: ETH0 I1122 00:12:26.443845 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:12:26.443860 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:12:29.450131 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:12:29.450212 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:12:29.450241 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:12:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:12:36.413260 3798194 memory.go:184] no items to output this cycle I1122 00:12:36.413416 3798194 cpu.go:275] no items to output this cycle E1122 00:12:46.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:12:46.413264 3798194 memory.go:184] no items to output this cycle I1122 00:12:46.413406 3798194 cpu.go:275] no items to output this cycle E1122 00:12:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:12:56.411950 3798194 memory.go:191] Add success. W1122 00:12:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:12:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:12:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:12:56.412357 3798194 cpu.go:282] Add success. I1122 00:12:56.441824 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:12:56.442348 3798194 net.go:642] Add success. I1122 00:12:56.446681 3798194 net.go:778] primary dev: ETH0 I1122 00:12:56.446697 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:12:56.446710 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:12:57.450605 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:12:57.450810 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:12:57.450821 3798194 disk_worker.go:723] disk space is not compliant W1122 00:12:57.450824 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:12:57.459224 3798194 disk_worker.go:509] system disk:vda1 I1122 00:12:57.459269 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:12:58.446386 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:12:59.450138 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:12:59.450228 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:12:59.450266 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:12:59.470731 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:13:00.656300 3798194 disk_info.go:125] begin check local disk info of client I1122 00:13:00.660225 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:13:00.660235 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000451080 0xc0004510c0] E1122 00:13:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:13:06.412256 3798194 memory.go:184] no items to output this cycle I1122 00:13:06.412404 3798194 cpu.go:275] no items to output this cycle E1122 00:13:16.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:13:16.412252 3798194 memory.go:184] no items to output this cycle I1122 00:13:16.412399 3798194 cpu.go:275] no items to output this cycle E1122 00:13:26.413169 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:13:26.413201 3798194 memory.go:191] Add success. I1122 00:13:26.417662 3798194 cpu.go:282] Add success. I1122 00:13:26.442189 3798194 net.go:642] Add success. I1122 00:13:26.443825 3798194 net.go:778] primary dev: ETH0 I1122 00:13:26.443839 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:13:26.443852 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:13:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:13:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:13:29.450144 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:13:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:13:36.411981 3798194 memory.go:184] no items to output this cycle I1122 00:13:36.412136 3798194 cpu.go:275] no items to output this cycle E1122 00:13:46.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:13:46.411989 3798194 memory.go:184] no items to output this cycle I1122 00:13:46.412157 3798194 cpu.go:275] no items to output this cycle E1122 00:13:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:13:56.411948 3798194 memory.go:191] Add success. W1122 00:13:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:13:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:13:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:13:56.412332 3798194 cpu.go:282] Add success. I1122 00:13:56.440961 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:13:56.441559 3798194 net.go:642] Add success. I1122 00:13:56.445787 3798194 net.go:778] primary dev: ETH0 I1122 00:13:56.445800 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:13:56.445813 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:13:57.451066 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:13:57.451243 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:13:57.451300 3798194 disk_worker.go:723] disk space is not compliant W1122 00:13:57.451303 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:13:57.460226 3798194 disk_worker.go:509] system disk:vda1 I1122 00:13:57.460265 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:13:58.448159 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:13:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:13:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:13:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:13:59.470666 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:14:00.660341 3798194 disk_info.go:125] begin check local disk info of client I1122 00:14:00.664186 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:14:00.664195 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e040 0xc00030e080] E1122 00:14:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:14:06.412258 3798194 memory.go:184] no items to output this cycle I1122 00:14:06.412399 3798194 cpu.go:275] no items to output this cycle E1122 00:14:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:14:16.411983 3798194 memory.go:184] no items to output this cycle I1122 00:14:16.412144 3798194 cpu.go:275] no items to output this cycle E1122 00:14:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:14:26.411950 3798194 memory.go:191] Add success. I1122 00:14:26.412302 3798194 cpu.go:282] Add success. I1122 00:14:26.441921 3798194 net.go:642] Add success. I1122 00:14:26.446239 3798194 net.go:778] primary dev: ETH0 I1122 00:14:26.446254 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:14:26.446266 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:14:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:14:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:14:29.450129 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:14:36.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:14:36.413262 3798194 memory.go:184] no items to output this cycle I1122 00:14:36.413404 3798194 cpu.go:275] no items to output this cycle E1122 00:14:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:14:46.413259 3798194 memory.go:184] no items to output this cycle I1122 00:14:46.413403 3798194 cpu.go:275] no items to output this cycle E1122 00:14:56.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:14:56.412279 3798194 memory.go:191] Add success. W1122 00:14:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:14:56.412319 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:14:56.412322 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:14:56.416801 3798194 cpu.go:282] Add success. I1122 00:14:56.441198 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:14:56.441754 3798194 net.go:642] Add success. I1122 00:14:56.443369 3798194 net.go:778] primary dev: ETH0 I1122 00:14:56.443384 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:14:56.443397 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:14:57.450615 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:14:57.450816 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:14:57.450828 3798194 disk_worker.go:723] disk space is not compliant W1122 00:14:57.450831 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:14:57.459175 3798194 disk_worker.go:509] system disk:vda1 I1122 00:14:57.459218 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:14:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:14:59.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:14:59.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:14:59.450489 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:14:59.471029 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:15:00.664291 3798194 disk_info.go:125] begin check local disk info of client I1122 00:15:00.668144 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:15:00.668153 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00044edc0 0xc00044ee00] E1122 00:15:06.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:15:06.412882 3798194 memory.go:184] no items to output this cycle I1122 00:15:06.413026 3798194 cpu.go:275] no items to output this cycle E1122 00:15:16.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:15:16.411994 3798194 memory.go:184] no items to output this cycle I1122 00:15:16.412149 3798194 cpu.go:275] no items to output this cycle I1122 00:15:17.581397 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:15:17.581405 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:15:26.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:15:26.413281 3798194 memory.go:191] Add success. I1122 00:15:26.417629 3798194 cpu.go:282] Add success. I1122 00:15:26.442168 3798194 net.go:642] Add success. I1122 00:15:26.443937 3798194 net.go:778] primary dev: ETH0 I1122 00:15:26.443951 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:15:26.443964 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:15:29.449928 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:15:29.450012 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:15:29.450041 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:15:36.412847 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:15:36.412870 3798194 memory.go:184] no items to output this cycle I1122 00:15:36.413014 3798194 cpu.go:275] no items to output this cycle E1122 00:15:46.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:15:46.412254 3798194 memory.go:184] no items to output this cycle I1122 00:15:46.412399 3798194 cpu.go:275] no items to output this cycle E1122 00:15:56.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:15:56.412884 3798194 memory.go:191] Add success. W1122 00:15:56.412910 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:15:56.412923 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:15:56.412926 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:15:56.417553 3798194 cpu.go:282] Add success. I1122 00:15:56.443641 3798194 event_worker.go:152] Polling the log file for events... I1122 00:15:56.443860 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:15:56.444357 3798194 net.go:642] Add success. I1122 00:15:56.445877 3798194 net.go:778] primary dev: ETH0 I1122 00:15:56.445892 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:15:56.445904 3798194 net.go:659] Add eni metrics success, count: 7 E1122 00:15:57.451218 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 00:15:57.451232 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:15:57.451436 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:15:57.451445 3798194 disk_worker.go:723] disk space is not compliant W1122 00:15:57.451448 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:15:57.459787 3798194 disk_worker.go:509] system disk:vda1 I1122 00:15:57.459829 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 00:15:58.448037 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 00:15:58.448051 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 00:15:59.448736 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 00:15:59.449811 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:15:59.449861 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:15:59.449885 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:15:59.471406 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:16:00.669156 3798194 disk_info.go:125] begin check local disk info of client I1122 00:16:00.672888 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:16:00.672897 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000372540 0xc000372580] E1122 00:16:06.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:16:06.413258 3798194 memory.go:184] no items to output this cycle I1122 00:16:06.413403 3798194 cpu.go:275] no items to output this cycle E1122 00:16:16.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:16:16.413261 3798194 memory.go:184] no items to output this cycle I1122 00:16:16.413407 3798194 cpu.go:275] no items to output this cycle E1122 00:16:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:16:26.413265 3798194 memory.go:191] Add success. I1122 00:16:26.417829 3798194 cpu.go:282] Add success. I1122 00:16:26.441292 3798194 net.go:642] Add success. I1122 00:16:26.442992 3798194 net.go:778] primary dev: ETH0 I1122 00:16:26.443007 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:16:26.443019 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:16:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:16:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:16:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:16:36.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:16:36.411984 3798194 memory.go:184] no items to output this cycle I1122 00:16:36.412142 3798194 cpu.go:275] no items to output this cycle E1122 00:16:46.412151 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:16:46.412179 3798194 memory.go:184] no items to output this cycle I1122 00:16:46.412624 3798194 cpu.go:275] no items to output this cycle E1122 00:16:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:16:56.411955 3798194 memory.go:191] Add success. W1122 00:16:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:16:56.412006 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:16:56.412010 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:16:56.412386 3798194 cpu.go:282] Add success. I1122 00:16:56.441850 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:16:56.442438 3798194 net.go:642] Add success. I1122 00:16:56.447468 3798194 net.go:778] primary dev: ETH0 I1122 00:16:56.447483 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:16:56.447497 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:16:57.450705 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:16:57.450880 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:16:57.450937 3798194 disk_worker.go:723] disk space is not compliant W1122 00:16:57.450941 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:16:57.459920 3798194 disk_worker.go:509] system disk:vda1 I1122 00:16:57.459963 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:16:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:16:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:16:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:16:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:16:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:17:00.673000 3798194 disk_info.go:125] begin check local disk info of client I1122 00:17:00.676774 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:17:00.676783 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00044fe80 0xc00044fec0] E1122 00:17:06.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:17:06.413268 3798194 memory.go:184] no items to output this cycle I1122 00:17:06.413409 3798194 cpu.go:275] no items to output this cycle E1122 00:17:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:17:16.411983 3798194 memory.go:184] no items to output this cycle I1122 00:17:16.412139 3798194 cpu.go:275] no items to output this cycle E1122 00:17:26.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:17:26.411956 3798194 memory.go:191] Add success. I1122 00:17:26.412318 3798194 cpu.go:282] Add success. I1122 00:17:26.441982 3798194 net.go:642] Add success. I1122 00:17:26.446296 3798194 net.go:778] primary dev: ETH0 I1122 00:17:26.446311 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:17:26.446323 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:17:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:17:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:17:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:17:36.412097 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:17:36.412136 3798194 memory.go:184] no items to output this cycle I1122 00:17:36.412278 3798194 cpu.go:275] no items to output this cycle E1122 00:17:46.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:17:46.413249 3798194 memory.go:184] no items to output this cycle I1122 00:17:46.413395 3798194 cpu.go:275] no items to output this cycle E1122 00:17:56.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:17:56.413276 3798194 memory.go:191] Add success. W1122 00:17:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:17:56.413316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:17:56.413320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:17:56.417976 3798194 cpu.go:282] Add success. I1122 00:17:56.441330 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:17:56.441812 3798194 net.go:642] Add success. I1122 00:17:56.443435 3798194 net.go:778] primary dev: ETH0 I1122 00:17:56.443451 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:17:56.443480 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:17:57.449159 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:17:57.449359 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:17:57.449416 3798194 disk_worker.go:723] disk space is not compliant W1122 00:17:57.449420 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:17:57.451232 3798194 disk_worker.go:509] system disk:vda1 I1122 00:17:57.451273 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:17:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:17:59.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:17:59.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:17:59.450517 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:17:59.471023 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:18:00.676888 3798194 disk_info.go:125] begin check local disk info of client I1122 00:18:00.681104 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:18:00.681130 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071e80 0xc000071ec0] E1122 00:18:06.416244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:18:06.416268 3798194 memory.go:184] no items to output this cycle I1122 00:18:06.416404 3798194 cpu.go:275] no items to output this cycle E1122 00:18:16.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:18:16.413268 3798194 memory.go:184] no items to output this cycle I1122 00:18:16.413412 3798194 cpu.go:275] no items to output this cycle I1122 00:18:17.581574 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:18:17.581585 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:18:26.413250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:18:26.413283 3798194 memory.go:191] Add success. I1122 00:18:26.418292 3798194 cpu.go:282] Add success. I1122 00:18:26.441743 3798194 net.go:642] Add success. I1122 00:18:26.443278 3798194 net.go:778] primary dev: ETH0 I1122 00:18:26.443295 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:18:26.443309 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:18:29.450195 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:18:29.450276 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:18:29.450302 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:18:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:18:36.412254 3798194 memory.go:184] no items to output this cycle I1122 00:18:36.412394 3798194 cpu.go:275] no items to output this cycle E1122 00:18:46.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:18:46.412260 3798194 memory.go:184] no items to output this cycle I1122 00:18:46.412410 3798194 cpu.go:275] no items to output this cycle E1122 00:18:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:18:56.411945 3798194 memory.go:191] Add success. W1122 00:18:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:18:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:18:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:18:56.412336 3798194 cpu.go:282] Add success. I1122 00:18:56.441824 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:18:56.442422 3798194 net.go:642] Add success. I1122 00:18:56.446567 3798194 net.go:778] primary dev: ETH0 I1122 00:18:56.446582 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:18:56.446596 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:18:57.450819 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:18:57.451023 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:18:57.451032 3798194 disk_worker.go:723] disk space is not compliant W1122 00:18:57.451036 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:18:57.459775 3798194 disk_worker.go:509] system disk:vda1 I1122 00:18:57.459818 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:18:58.446359 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:18:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:18:59.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:18:59.450489 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:18:59.471001 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:19:00.682136 3798194 disk_info.go:125] begin check local disk info of client I1122 00:19:00.685969 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:19:00.685978 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f1bc0 0xc0003f1c00] E1122 00:19:06.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:19:06.413254 3798194 memory.go:184] no items to output this cycle I1122 00:19:06.413395 3798194 cpu.go:275] no items to output this cycle E1122 00:19:16.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:19:16.411990 3798194 memory.go:184] no items to output this cycle I1122 00:19:16.412151 3798194 cpu.go:275] no items to output this cycle E1122 00:19:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:19:26.411952 3798194 memory.go:191] Add success. I1122 00:19:26.412338 3798194 cpu.go:282] Add success. I1122 00:19:26.441944 3798194 net.go:642] Add success. I1122 00:19:26.446093 3798194 net.go:778] primary dev: ETH0 I1122 00:19:26.446124 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:19:26.446139 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:19:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:19:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:19:29.450145 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:19:36.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:19:36.412264 3798194 memory.go:184] no items to output this cycle I1122 00:19:36.412405 3798194 cpu.go:275] no items to output this cycle E1122 00:19:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:19:46.413253 3798194 memory.go:184] no items to output this cycle I1122 00:19:46.413408 3798194 cpu.go:275] no items to output this cycle E1122 00:19:56.411908 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:19:56.411943 3798194 memory.go:191] Add success. W1122 00:19:56.411971 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:19:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:19:56.411987 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:19:56.412335 3798194 cpu.go:282] Add success. I1122 00:19:56.441797 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:19:56.442393 3798194 net.go:642] Add success. I1122 00:19:56.447026 3798194 net.go:778] primary dev: ETH0 I1122 00:19:56.447041 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:19:56.447056 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:19:57.443679 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:19:57.445466 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:19:57.445481 3798194 disk_worker.go:723] disk space is not compliant W1122 00:19:57.445484 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:19:57.447583 3798194 disk_worker.go:509] system disk:vda1 I1122 00:19:57.447622 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:19:58.446351 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:19:59.450459 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:19:59.450537 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:19:59.450561 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:19:59.471082 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:20:00.686083 3798194 disk_info.go:125] begin check local disk info of client I1122 00:20:00.689900 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:20:00.689909 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000336340 0xc000336380] E1122 00:20:06.413263 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:20:06.413285 3798194 memory.go:184] no items to output this cycle I1122 00:20:06.413433 3798194 cpu.go:275] no items to output this cycle E1122 00:20:16.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:20:16.413261 3798194 memory.go:184] no items to output this cycle I1122 00:20:16.413410 3798194 cpu.go:275] no items to output this cycle E1122 00:20:26.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:20:26.411947 3798194 memory.go:191] Add success. I1122 00:20:26.412312 3798194 cpu.go:282] Add success. I1122 00:20:26.441919 3798194 net.go:642] Add success. I1122 00:20:26.446540 3798194 net.go:778] primary dev: ETH0 I1122 00:20:26.446556 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:20:26.446568 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:20:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:20:29.450119 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:20:29.450151 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:20:36.412328 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:20:36.412351 3798194 memory.go:184] no items to output this cycle I1122 00:20:36.412502 3798194 cpu.go:275] no items to output this cycle E1122 00:20:46.411699 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:20:46.411721 3798194 memory.go:184] no items to output this cycle I1122 00:20:46.411876 3798194 cpu.go:275] no items to output this cycle E1122 00:20:56.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:20:56.413271 3798194 memory.go:191] Add success. W1122 00:20:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:20:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:20:56.413312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:20:56.417652 3798194 cpu.go:282] Add success. I1122 00:20:56.443816 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:20:56.444316 3798194 net.go:642] Add success. I1122 00:20:56.445722 3798194 net.go:778] primary dev: ETH0 I1122 00:20:56.445737 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:20:56.445751 3798194 net.go:659] Add eni metrics success, count: 7 E1122 00:20:57.450900 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 00:20:57.450913 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:20:57.451103 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:20:57.451134 3798194 disk_worker.go:723] disk space is not compliant W1122 00:20:57.451138 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:20:57.459892 3798194 disk_worker.go:509] system disk:vda1 I1122 00:20:57.459933 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 00:20:58.448465 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 00:20:58.448479 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 00:20:59.448577 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 00:20:59.449684 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:20:59.449748 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:20:59.449772 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:20:59.471378 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:21:00.690015 3798194 disk_info.go:125] begin check local disk info of client I1122 00:21:00.693786 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:21:00.693795 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00025e340 0xc00025e380] E1122 00:21:06.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:21:06.413262 3798194 memory.go:184] no items to output this cycle I1122 00:21:06.413408 3798194 cpu.go:275] no items to output this cycle E1122 00:21:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:21:16.413258 3798194 memory.go:184] no items to output this cycle I1122 00:21:16.413407 3798194 cpu.go:275] no items to output this cycle I1122 00:21:17.581744 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:21:17.581753 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:21:26.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:21:26.413267 3798194 memory.go:191] Add success. I1122 00:21:26.418192 3798194 cpu.go:282] Add success. I1122 00:21:26.441687 3798194 net.go:642] Add success. I1122 00:21:26.443259 3798194 net.go:778] primary dev: ETH0 I1122 00:21:26.443273 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:21:26.443286 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:21:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:21:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:21:29.450501 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:21:36.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:21:36.413257 3798194 memory.go:184] no items to output this cycle I1122 00:21:36.413407 3798194 cpu.go:275] no items to output this cycle E1122 00:21:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:21:46.412262 3798194 memory.go:184] no items to output this cycle I1122 00:21:46.412405 3798194 cpu.go:275] no items to output this cycle E1122 00:21:56.412223 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:21:56.412256 3798194 memory.go:191] Add success. W1122 00:21:56.412284 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:21:56.412296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:21:56.412299 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:21:56.416663 3798194 cpu.go:282] Add success. I1122 00:21:56.440948 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:21:56.441519 3798194 net.go:642] Add success. I1122 00:21:56.443203 3798194 net.go:778] primary dev: ETH0 I1122 00:21:56.443219 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:21:56.443232 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:21:57.450973 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:21:57.451205 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:21:57.451216 3798194 disk_worker.go:723] disk space is not compliant W1122 00:21:57.451219 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:21:57.459934 3798194 disk_worker.go:509] system disk:vda1 I1122 00:21:57.459974 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:21:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:21:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:21:59.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:21:59.450509 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:21:59.471005 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:22:00.693899 3798194 disk_info.go:125] begin check local disk info of client I1122 00:22:00.697756 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:22:00.697766 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030fb40 0xc00030fb80] E1122 00:22:06.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:22:06.413263 3798194 memory.go:184] no items to output this cycle I1122 00:22:06.413406 3798194 cpu.go:275] no items to output this cycle E1122 00:22:16.412000 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:22:16.412032 3798194 memory.go:184] no items to output this cycle I1122 00:22:16.412201 3798194 cpu.go:275] no items to output this cycle E1122 00:22:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:22:26.411955 3798194 memory.go:191] Add success. I1122 00:22:26.412348 3798194 cpu.go:282] Add success. I1122 00:22:26.441989 3798194 net.go:642] Add success. I1122 00:22:26.445988 3798194 net.go:778] primary dev: ETH0 I1122 00:22:26.446004 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:22:26.446016 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:22:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:22:29.450490 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:22:29.450518 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:22:36.413212 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:22:36.413235 3798194 memory.go:184] no items to output this cycle I1122 00:22:36.413384 3798194 cpu.go:275] no items to output this cycle E1122 00:22:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:22:46.413254 3798194 memory.go:184] no items to output this cycle I1122 00:22:46.413395 3798194 cpu.go:275] no items to output this cycle E1122 00:22:56.413066 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:22:56.413106 3798194 memory.go:191] Add success. W1122 00:22:56.413157 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:22:56.413168 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:22:56.413171 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:22:56.417428 3798194 cpu.go:282] Add success. I1122 00:22:56.441854 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:22:56.442439 3798194 net.go:642] Add success. I1122 00:22:56.443952 3798194 net.go:778] primary dev: ETH0 I1122 00:22:56.443966 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:22:56.443980 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:22:57.451690 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:22:57.451865 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:22:57.451921 3798194 disk_worker.go:723] disk space is not compliant W1122 00:22:57.451924 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:22:57.465208 3798194 disk_worker.go:509] system disk:vda1 I1122 00:22:57.465247 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:22:58.446375 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:22:59.450476 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:22:59.450558 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:22:59.450583 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:22:59.471104 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:23:00.697877 3798194 disk_info.go:125] begin check local disk info of client I1122 00:23:00.703757 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:23:00.703766 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d9680 0xc0003d96c0] E1122 00:23:06.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:23:06.413269 3798194 memory.go:184] no items to output this cycle I1122 00:23:06.413420 3798194 cpu.go:275] no items to output this cycle E1122 00:23:16.411902 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:23:16.411924 3798194 memory.go:184] no items to output this cycle I1122 00:23:16.412075 3798194 cpu.go:275] no items to output this cycle E1122 00:23:26.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:23:26.412897 3798194 memory.go:191] Add success. I1122 00:23:26.417602 3798194 cpu.go:282] Add success. I1122 00:23:26.442145 3798194 net.go:642] Add success. I1122 00:23:26.443617 3798194 net.go:778] primary dev: ETH0 I1122 00:23:26.443631 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:23:26.443645 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:23:29.450196 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:23:29.450290 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:23:29.450316 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:23:36.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:23:36.412874 3798194 memory.go:184] no items to output this cycle I1122 00:23:36.413014 3798194 cpu.go:275] no items to output this cycle E1122 00:23:46.412980 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:23:46.413004 3798194 memory.go:184] no items to output this cycle I1122 00:23:46.413180 3798194 cpu.go:275] no items to output this cycle E1122 00:23:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:23:56.411955 3798194 memory.go:191] Add success. W1122 00:23:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:23:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:23:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:23:56.412331 3798194 cpu.go:282] Add success. I1122 00:23:56.441823 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:23:56.442322 3798194 net.go:642] Add success. I1122 00:23:56.446422 3798194 net.go:778] primary dev: ETH0 I1122 00:23:56.446437 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:23:56.446449 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:23:57.450973 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:23:57.451211 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:23:57.451222 3798194 disk_worker.go:723] disk space is not compliant W1122 00:23:57.451225 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:23:57.459998 3798194 disk_worker.go:509] system disk:vda1 I1122 00:23:57.460040 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:23:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:23:59.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:23:59.450124 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:23:59.450150 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:23:59.471667 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:24:00.705137 3798194 disk_info.go:125] begin check local disk info of client I1122 00:24:00.708984 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:24:00.708994 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b4340 0xc0003b4380] E1122 00:24:06.412780 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:24:06.412802 3798194 memory.go:184] no items to output this cycle I1122 00:24:06.412950 3798194 cpu.go:275] no items to output this cycle E1122 00:24:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:24:16.411986 3798194 memory.go:184] no items to output this cycle I1122 00:24:16.412149 3798194 cpu.go:275] no items to output this cycle I1122 00:24:17.581919 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:24:17.581928 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:24:26.412846 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:24:26.412880 3798194 memory.go:191] Add success. I1122 00:24:26.417209 3798194 cpu.go:282] Add success. I1122 00:24:26.442006 3798194 net.go:642] Add success. I1122 00:24:26.445290 3798194 net.go:778] primary dev: ETH0 I1122 00:24:26.445306 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:24:26.445326 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:24:29.450213 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:24:29.450302 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:24:29.450332 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:24:36.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:24:36.412877 3798194 memory.go:184] no items to output this cycle I1122 00:24:36.413019 3798194 cpu.go:275] no items to output this cycle E1122 00:24:46.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:24:46.413259 3798194 memory.go:184] no items to output this cycle I1122 00:24:46.413413 3798194 cpu.go:275] no items to output this cycle E1122 00:24:56.411195 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:24:56.411228 3798194 memory.go:191] Add success. W1122 00:24:56.411256 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:24:56.411269 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:24:56.411273 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:24:56.415300 3798194 cpu.go:282] Add success. I1122 00:24:56.441713 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:24:56.442212 3798194 net.go:642] Add success. I1122 00:24:56.443788 3798194 net.go:778] primary dev: ETH0 I1122 00:24:56.443804 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:24:56.443818 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:24:57.451175 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:24:57.451331 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:24:57.451382 3798194 disk_worker.go:723] disk space is not compliant W1122 00:24:57.451386 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:24:57.460082 3798194 disk_worker.go:509] system disk:vda1 I1122 00:24:57.460142 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:24:58.446628 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:24:59.449677 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:24:59.449757 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:24:59.449782 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:24:59.471387 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:25:00.709098 3798194 disk_info.go:125] begin check local disk info of client I1122 00:25:00.712931 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:25:00.712941 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000480780 0xc0004807c0] E1122 00:25:06.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:25:06.413259 3798194 memory.go:184] no items to output this cycle I1122 00:25:06.413403 3798194 cpu.go:275] no items to output this cycle E1122 00:25:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:25:16.413256 3798194 memory.go:184] no items to output this cycle I1122 00:25:16.413401 3798194 cpu.go:275] no items to output this cycle E1122 00:25:26.411657 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:25:26.411690 3798194 memory.go:191] Add success. I1122 00:25:26.415872 3798194 cpu.go:282] Add success. I1122 00:25:26.441430 3798194 net.go:642] Add success. I1122 00:25:26.443034 3798194 net.go:778] primary dev: ETH0 I1122 00:25:26.443051 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:25:26.443065 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:25:29.449615 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:25:29.449702 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:25:29.449728 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:25:36.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:25:36.413261 3798194 memory.go:184] no items to output this cycle I1122 00:25:36.413413 3798194 cpu.go:275] no items to output this cycle E1122 00:25:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:25:46.412254 3798194 memory.go:184] no items to output this cycle I1122 00:25:46.412401 3798194 cpu.go:275] no items to output this cycle E1122 00:25:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:25:56.413270 3798194 memory.go:191] Add success. W1122 00:25:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:25:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:25:56.413311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:25:56.417265 3798194 cpu.go:282] Add success. I1122 00:25:56.441438 3798194 event_worker.go:152] Polling the log file for events... I1122 00:25:56.444544 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:25:56.445446 3798194 net.go:642] Add success. I1122 00:25:56.446930 3798194 net.go:778] primary dev: ETH0 I1122 00:25:56.446951 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:25:56.446965 3798194 net.go:659] Add eni metrics success, count: 7 E1122 00:25:57.450987 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 00:25:57.451001 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:25:57.451182 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:25:57.451237 3798194 disk_worker.go:723] disk space is not compliant W1122 00:25:57.451240 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:25:57.460033 3798194 disk_worker.go:509] system disk:vda1 I1122 00:25:57.460083 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 00:25:58.448704 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 00:25:58.448718 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 00:25:59.448577 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 00:25:59.449656 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:25:59.449710 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:25:59.449730 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:25:59.471195 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:26:00.713145 3798194 disk_info.go:125] begin check local disk info of client I1122 00:26:00.716983 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:26:00.716992 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004e43c0 0xc0004e4400] E1122 00:26:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:26:06.412255 3798194 memory.go:184] no items to output this cycle I1122 00:26:06.412398 3798194 cpu.go:275] no items to output this cycle E1122 00:26:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:26:16.413256 3798194 memory.go:184] no items to output this cycle I1122 00:26:16.413407 3798194 cpu.go:275] no items to output this cycle E1122 00:26:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:26:26.413266 3798194 memory.go:191] Add success. I1122 00:26:26.425643 3798194 cpu.go:282] Add success. I1122 00:26:26.442163 3798194 net.go:642] Add success. I1122 00:26:26.443795 3798194 net.go:778] primary dev: ETH0 I1122 00:26:26.443809 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:26:26.443824 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:26:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:26:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:26:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:26:36.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:26:36.411970 3798194 memory.go:184] no items to output this cycle I1122 00:26:36.412127 3798194 cpu.go:275] no items to output this cycle E1122 00:26:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:26:46.411975 3798194 memory.go:184] no items to output this cycle I1122 00:26:46.412130 3798194 cpu.go:275] no items to output this cycle E1122 00:26:56.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:26:56.413272 3798194 memory.go:191] Add success. W1122 00:26:56.413299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:26:56.413310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:26:56.413313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:26:56.417428 3798194 cpu.go:282] Add success. I1122 00:26:56.441824 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:26:56.442345 3798194 net.go:642] Add success. I1122 00:26:56.443831 3798194 net.go:778] primary dev: ETH0 I1122 00:26:56.443845 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:26:56.443856 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:26:57.451159 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:26:57.451317 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:26:57.451366 3798194 disk_worker.go:723] disk space is not compliant W1122 00:26:57.451370 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:26:57.459722 3798194 disk_worker.go:509] system disk:vda1 I1122 00:26:57.459761 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:26:58.446387 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:26:59.450500 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:26:59.450582 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:26:59.450611 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:26:59.471128 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:27:00.717135 3798194 disk_info.go:125] begin check local disk info of client I1122 00:27:00.722690 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:27:00.722700 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000416980 0xc0004169c0] E1122 00:27:06.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:27:06.413260 3798194 memory.go:184] no items to output this cycle I1122 00:27:06.413406 3798194 cpu.go:275] no items to output this cycle E1122 00:27:16.412223 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:27:16.412245 3798194 memory.go:184] no items to output this cycle I1122 00:27:16.412676 3798194 cpu.go:275] no items to output this cycle I1122 00:27:17.582084 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:27:17.582093 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:27:26.411437 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:27:26.411469 3798194 memory.go:191] Add success. I1122 00:27:26.411832 3798194 cpu.go:282] Add success. I1122 00:27:26.441450 3798194 net.go:642] Add success. I1122 00:27:26.445483 3798194 net.go:778] primary dev: ETH0 I1122 00:27:26.445498 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:27:26.445511 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:27:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:27:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:27:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:27:36.412806 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:27:36.412828 3798194 memory.go:184] no items to output this cycle I1122 00:27:36.412968 3798194 cpu.go:275] no items to output this cycle E1122 00:27:46.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:27:46.412255 3798194 memory.go:184] no items to output this cycle I1122 00:27:46.412399 3798194 cpu.go:275] no items to output this cycle E1122 00:27:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:27:56.411946 3798194 memory.go:191] Add success. W1122 00:27:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:27:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:27:56.411993 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:27:56.412335 3798194 cpu.go:282] Add success. I1122 00:27:56.441836 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:27:56.442427 3798194 net.go:642] Add success. I1122 00:27:56.446578 3798194 net.go:778] primary dev: ETH0 I1122 00:27:56.446593 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:27:56.446606 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:27:57.450747 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:27:57.450967 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:27:57.450979 3798194 disk_worker.go:723] disk space is not compliant W1122 00:27:57.450982 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:27:57.459956 3798194 disk_worker.go:509] system disk:vda1 I1122 00:27:57.459996 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:27:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:27:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:27:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:27:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:27:59.471639 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:28:00.723145 3798194 disk_info.go:125] begin check local disk info of client I1122 00:28:00.727057 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:28:00.727067 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002ffc40 0xc0002ffc80] E1122 00:28:06.412440 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:28:06.412473 3798194 memory.go:184] no items to output this cycle I1122 00:28:06.412684 3798194 cpu.go:275] no items to output this cycle E1122 00:28:16.412655 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:28:16.412680 3798194 memory.go:184] no items to output this cycle I1122 00:28:16.412830 3798194 cpu.go:275] no items to output this cycle E1122 00:28:26.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:28:26.412890 3798194 memory.go:191] Add success. I1122 00:28:26.417689 3798194 cpu.go:282] Add success. I1122 00:28:26.442222 3798194 net.go:642] Add success. I1122 00:28:26.443776 3798194 net.go:778] primary dev: ETH0 I1122 00:28:26.443790 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:28:26.443804 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:28:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:28:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:28:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:28:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:28:36.413261 3798194 memory.go:184] no items to output this cycle I1122 00:28:36.413405 3798194 cpu.go:275] no items to output this cycle E1122 00:28:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:28:46.413261 3798194 memory.go:184] no items to output this cycle I1122 00:28:46.413407 3798194 cpu.go:275] no items to output this cycle E1122 00:28:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:28:56.412274 3798194 memory.go:191] Add success. W1122 00:28:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:28:56.412314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:28:56.412317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:28:56.416558 3798194 cpu.go:282] Add success. I1122 00:28:56.441229 3798194 net.go:642] Add success. I1122 00:28:56.443193 3798194 net.go:778] primary dev: ETH0 I1122 00:28:56.443207 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:28:56.443220 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:28:56.441589 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:28:57.445561 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:28:57.445766 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:28:57.445832 3798194 disk_worker.go:723] disk space is not compliant W1122 00:28:57.445835 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:28:57.448049 3798194 disk_worker.go:509] system disk:vda1 I1122 00:28:57.448090 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:28:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:28:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:28:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:28:59.450147 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:28:59.471646 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:29:00.727167 3798194 disk_info.go:125] begin check local disk info of client I1122 00:29:00.731061 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:29:00.731073 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e1580 0xc0002e15c0] E1122 00:29:06.413262 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:29:06.413284 3798194 memory.go:184] no items to output this cycle I1122 00:29:06.413440 3798194 cpu.go:275] no items to output this cycle E1122 00:29:16.413264 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:29:16.413285 3798194 memory.go:184] no items to output this cycle I1122 00:29:16.413433 3798194 cpu.go:275] no items to output this cycle E1122 00:29:26.412828 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:29:26.412862 3798194 memory.go:191] Add success. I1122 00:29:26.417126 3798194 cpu.go:282] Add success. I1122 00:29:26.442576 3798194 net.go:642] Add success. I1122 00:29:26.444186 3798194 net.go:778] primary dev: ETH0 I1122 00:29:26.444210 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:29:26.444229 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:29:29.450010 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:29:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:29:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:29:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:29:36.413263 3798194 memory.go:184] no items to output this cycle I1122 00:29:36.413403 3798194 cpu.go:275] no items to output this cycle E1122 00:29:46.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:29:46.412886 3798194 memory.go:184] no items to output this cycle I1122 00:29:46.413031 3798194 cpu.go:275] no items to output this cycle E1122 00:29:56.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:29:56.412888 3798194 memory.go:191] Add success. W1122 00:29:56.412922 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:29:56.412933 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:29:56.412937 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:29:56.417377 3798194 cpu.go:282] Add success. I1122 00:29:56.441810 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:29:56.442326 3798194 net.go:642] Add success. I1122 00:29:56.443762 3798194 net.go:778] primary dev: ETH0 I1122 00:29:56.443777 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:29:56.443791 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:29:57.451272 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:29:57.451482 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:29:57.451494 3798194 disk_worker.go:723] disk space is not compliant W1122 00:29:57.451497 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:29:57.459881 3798194 disk_worker.go:509] system disk:vda1 I1122 00:29:57.459923 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:29:58.446375 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:29:59.450476 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:29:59.450552 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:29:59.450581 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:29:59.471104 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:30:00.731190 3798194 disk_info.go:125] begin check local disk info of client I1122 00:30:00.735972 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:30:00.735984 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000500040 0xc000500080] E1122 00:30:06.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:30:06.413263 3798194 memory.go:184] no items to output this cycle I1122 00:30:06.413411 3798194 cpu.go:275] no items to output this cycle E1122 00:30:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:30:16.413264 3798194 memory.go:184] no items to output this cycle I1122 00:30:16.413409 3798194 cpu.go:275] no items to output this cycle I1122 00:30:17.582220 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:30:17.582229 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:30:26.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:30:26.412892 3798194 memory.go:191] Add success. I1122 00:30:26.417274 3798194 cpu.go:282] Add success. I1122 00:30:26.441750 3798194 net.go:642] Add success. I1122 00:30:26.443234 3798194 net.go:778] primary dev: ETH0 I1122 00:30:26.443250 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:30:26.443263 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:30:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:30:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:30:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:30:36.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:30:36.412248 3798194 memory.go:184] no items to output this cycle I1122 00:30:36.412389 3798194 cpu.go:275] no items to output this cycle E1122 00:30:46.413284 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:30:46.413305 3798194 memory.go:184] no items to output this cycle I1122 00:30:46.413455 3798194 cpu.go:275] no items to output this cycle E1122 00:30:56.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:30:56.412258 3798194 memory.go:191] Add success. W1122 00:30:56.412285 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:30:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:30:56.412309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:30:56.416589 3798194 cpu.go:282] Add success. I1122 00:30:56.442633 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:30:56.443107 3798194 net.go:642] Add success. I1122 00:30:56.445123 3798194 net.go:778] primary dev: ETH0 I1122 00:30:56.445142 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:30:56.445155 3798194 net.go:659] Add eni metrics success, count: 7 E1122 00:30:57.445589 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 00:30:57.445614 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:30:57.445854 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:30:57.445865 3798194 disk_worker.go:723] disk space is not compliant W1122 00:30:57.445868 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:30:57.450868 3798194 disk_worker.go:509] system disk:vda1 I1122 00:30:57.450911 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 00:30:58.448093 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 00:30:58.448126 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 00:30:59.449001 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 00:30:59.450081 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:30:59.450147 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:30:59.450169 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:30:59.471651 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:31:00.736144 3798194 disk_info.go:125] begin check local disk info of client I1122 00:31:00.741192 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:31:00.741202 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000353580 0xc0003535c0] E1122 00:31:06.412419 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:31:06.412440 3798194 memory.go:184] no items to output this cycle I1122 00:31:06.412582 3798194 cpu.go:275] no items to output this cycle E1122 00:31:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:31:16.413264 3798194 memory.go:184] no items to output this cycle I1122 00:31:16.413412 3798194 cpu.go:275] no items to output this cycle E1122 00:31:26.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:31:26.412880 3798194 memory.go:191] Add success. I1122 00:31:26.417775 3798194 cpu.go:282] Add success. I1122 00:31:26.441261 3798194 net.go:642] Add success. I1122 00:31:26.442817 3798194 net.go:778] primary dev: ETH0 I1122 00:31:26.442833 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:31:26.442848 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:31:29.450634 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:31:29.450712 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:31:29.450741 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:31:36.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:31:36.413247 3798194 memory.go:184] no items to output this cycle I1122 00:31:36.413574 3798194 cpu.go:275] no items to output this cycle E1122 00:31:46.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:31:46.412882 3798194 memory.go:184] no items to output this cycle I1122 00:31:46.413028 3798194 cpu.go:275] no items to output this cycle E1122 00:31:56.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:31:56.412270 3798194 memory.go:191] Add success. W1122 00:31:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:31:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:31:56.412312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:31:56.416738 3798194 cpu.go:282] Add success. I1122 00:31:56.441101 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:31:56.441593 3798194 net.go:642] Add success. I1122 00:31:56.443150 3798194 net.go:778] primary dev: ETH0 I1122 00:31:56.443166 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:31:56.443179 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:31:57.450518 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:31:57.450685 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:31:57.450739 3798194 disk_worker.go:723] disk space is not compliant W1122 00:31:57.450742 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:31:57.459141 3798194 disk_worker.go:509] system disk:vda1 I1122 00:31:57.459182 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:31:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:31:59.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:31:59.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:31:59.450500 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:31:59.471018 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:32:00.742144 3798194 disk_info.go:125] begin check local disk info of client I1122 00:32:00.745973 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:32:00.745982 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d0d80 0xc0003d0e40] E1122 00:32:06.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:32:06.412876 3798194 memory.go:184] no items to output this cycle I1122 00:32:06.413025 3798194 cpu.go:275] no items to output this cycle E1122 00:32:16.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:32:16.412873 3798194 memory.go:184] no items to output this cycle I1122 00:32:16.413020 3798194 cpu.go:275] no items to output this cycle E1122 00:32:26.412839 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:32:26.412872 3798194 memory.go:191] Add success. I1122 00:32:26.417200 3798194 cpu.go:282] Add success. I1122 00:32:26.441669 3798194 net.go:642] Add success. I1122 00:32:26.443102 3798194 net.go:778] primary dev: ETH0 I1122 00:32:26.443131 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:32:26.443146 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:32:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:32:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:32:29.450496 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:32:36.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:32:36.413254 3798194 memory.go:184] no items to output this cycle I1122 00:32:36.413403 3798194 cpu.go:275] no items to output this cycle E1122 00:32:46.412996 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:32:46.413017 3798194 memory.go:184] no items to output this cycle I1122 00:32:46.413176 3798194 cpu.go:275] no items to output this cycle E1122 00:32:56.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:32:56.413270 3798194 memory.go:191] Add success. W1122 00:32:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:32:56.413315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:32:56.413318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:32:56.418108 3798194 cpu.go:282] Add success. I1122 00:32:56.441419 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:32:56.441891 3798194 net.go:642] Add success. I1122 00:32:56.443642 3798194 net.go:778] primary dev: ETH0 I1122 00:32:56.443657 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:32:56.443669 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:32:57.445504 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:32:57.445764 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:32:57.445778 3798194 disk_worker.go:723] disk space is not compliant W1122 00:32:57.445781 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:32:57.448015 3798194 disk_worker.go:509] system disk:vda1 I1122 00:32:57.448069 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:32:58.445984 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:32:59.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:32:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:32:59.450149 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:32:59.470689 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:33:00.746086 3798194 disk_info.go:125] begin check local disk info of client I1122 00:33:00.750014 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:33:00.750030 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047e880 0xc00047e8c0] E1122 00:33:06.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:33:06.413255 3798194 memory.go:184] no items to output this cycle I1122 00:33:06.413398 3798194 cpu.go:275] no items to output this cycle E1122 00:33:16.413590 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:33:16.413614 3798194 memory.go:184] no items to output this cycle I1122 00:33:16.413760 3798194 cpu.go:275] no items to output this cycle I1122 00:33:17.582413 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:33:17.582424 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:33:26.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:33:26.412890 3798194 memory.go:191] Add success. I1122 00:33:26.417322 3798194 cpu.go:282] Add success. I1122 00:33:26.441757 3798194 net.go:642] Add success. I1122 00:33:26.443266 3798194 net.go:778] primary dev: ETH0 I1122 00:33:26.443280 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:33:26.443292 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:33:29.450052 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:33:29.450137 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:33:29.450163 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:33:36.413223 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:33:36.413249 3798194 memory.go:184] no items to output this cycle I1122 00:33:36.413393 3798194 cpu.go:275] no items to output this cycle E1122 00:33:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:33:46.412258 3798194 memory.go:184] no items to output this cycle I1122 00:33:46.412404 3798194 cpu.go:275] no items to output this cycle E1122 00:33:56.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:33:56.413264 3798194 memory.go:191] Add success. W1122 00:33:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:33:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:33:56.413311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:33:56.418011 3798194 cpu.go:282] Add success. I1122 00:33:56.441425 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:33:56.441985 3798194 net.go:642] Add success. I1122 00:33:56.443464 3798194 net.go:778] primary dev: ETH0 I1122 00:33:56.443480 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:33:56.443492 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:33:57.446491 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:33:57.446766 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:33:57.446778 3798194 disk_worker.go:723] disk space is not compliant W1122 00:33:57.446781 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:33:57.449608 3798194 disk_worker.go:509] system disk:vda1 I1122 00:33:57.449648 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:33:58.446381 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:33:59.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:33:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:33:59.450504 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:33:59.471028 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:34:00.750140 3798194 disk_info.go:125] begin check local disk info of client I1122 00:34:00.754063 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:34:00.754072 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f0400 0xc0003f0440] E1122 00:34:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:34:06.412255 3798194 memory.go:184] no items to output this cycle I1122 00:34:06.412401 3798194 cpu.go:275] no items to output this cycle E1122 00:34:16.412333 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:34:16.412358 3798194 memory.go:184] no items to output this cycle I1122 00:34:16.412514 3798194 cpu.go:275] no items to output this cycle E1122 00:34:26.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:34:26.412262 3798194 memory.go:191] Add success. I1122 00:34:26.416393 3798194 cpu.go:282] Add success. I1122 00:34:26.442351 3798194 net.go:642] Add success. I1122 00:34:26.444023 3798194 net.go:778] primary dev: ETH0 I1122 00:34:26.444040 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:34:26.444138 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:34:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:34:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:34:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:34:36.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:34:36.412268 3798194 memory.go:184] no items to output this cycle I1122 00:34:36.412408 3798194 cpu.go:275] no items to output this cycle E1122 00:34:46.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:34:46.412266 3798194 memory.go:184] no items to output this cycle I1122 00:34:46.412409 3798194 cpu.go:275] no items to output this cycle E1122 00:34:56.412224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:34:56.412260 3798194 memory.go:191] Add success. W1122 00:34:56.412288 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:34:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:34:56.412303 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:34:56.416617 3798194 cpu.go:282] Add success. I1122 00:34:56.440968 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:34:56.441577 3798194 net.go:642] Add success. I1122 00:34:56.443226 3798194 net.go:778] primary dev: ETH0 I1122 00:34:56.443241 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:34:56.443254 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:34:57.450514 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:34:57.450718 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:34:57.450728 3798194 disk_worker.go:723] disk space is not compliant W1122 00:34:57.450731 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:34:57.459015 3798194 disk_worker.go:509] system disk:vda1 I1122 00:34:57.459056 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:34:58.446304 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:34:59.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:34:59.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:34:59.450506 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:34:59.471035 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:35:00.754174 3798194 disk_info.go:125] begin check local disk info of client I1122 00:35:00.758127 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:35:00.758137 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00045e840 0xc00045e880] E1122 00:35:06.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:35:06.412901 3798194 memory.go:184] no items to output this cycle I1122 00:35:06.413053 3798194 cpu.go:275] no items to output this cycle E1122 00:35:16.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:35:16.413249 3798194 memory.go:184] no items to output this cycle I1122 00:35:16.413392 3798194 cpu.go:275] no items to output this cycle E1122 00:35:26.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:35:26.412270 3798194 memory.go:191] Add success. I1122 00:35:26.416454 3798194 cpu.go:282] Add success. I1122 00:35:26.441950 3798194 net.go:642] Add success. I1122 00:35:26.443411 3798194 net.go:778] primary dev: ETH0 I1122 00:35:26.443427 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:35:26.443439 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:35:29.449960 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:35:29.450038 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:35:29.450072 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:35:36.413049 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:35:36.413070 3798194 memory.go:184] no items to output this cycle I1122 00:35:36.413225 3798194 cpu.go:275] no items to output this cycle E1122 00:35:46.413043 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:35:46.413071 3798194 memory.go:184] no items to output this cycle I1122 00:35:46.413233 3798194 cpu.go:275] no items to output this cycle E1122 00:35:56.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:35:56.412270 3798194 memory.go:191] Add success. W1122 00:35:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:35:56.412314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:35:56.412317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:35:56.416497 3798194 cpu.go:282] Add success. I1122 00:35:56.441666 3798194 event_worker.go:152] Polling the log file for events... I1122 00:35:56.445233 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:35:56.446216 3798194 net.go:642] Add success. I1122 00:35:56.447665 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795277156, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 00:35:56.447868 3798194 net.go:778] primary dev: ETH0 I1122 00:35:56.447886 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:35:56.447900 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:35:56.459151 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 00:35:57.447505 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 00:35:57.447520 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:35:57.447722 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:35:57.447801 3798194 disk_worker.go:723] disk space is not compliant W1122 00:35:57.447804 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:35:57.451627 3798194 disk_worker.go:509] system disk:vda1 I1122 00:35:57.451668 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 00:35:58.448657 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 00:35:58.448671 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:35:59.450157 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:35:59.450212 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:35:59.450230 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:35:59.450353 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 00:35:59.470723 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:36:00.758246 3798194 disk_info.go:125] begin check local disk info of client I1122 00:36:00.762048 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:36:00.762058 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004f8680 0xc0004f86c0] E1122 00:36:06.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:36:06.413265 3798194 memory.go:184] no items to output this cycle I1122 00:36:06.413417 3798194 cpu.go:275] no items to output this cycle E1122 00:36:16.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:36:16.413252 3798194 memory.go:184] no items to output this cycle I1122 00:36:16.413407 3798194 cpu.go:275] no items to output this cycle I1122 00:36:17.582593 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:36:17.582601 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:36:26.413425 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:36:26.413457 3798194 memory.go:191] Add success. I1122 00:36:26.417849 3798194 cpu.go:282] Add success. I1122 00:36:26.441186 3798194 net.go:642] Add success. I1122 00:36:26.442681 3798194 net.go:778] primary dev: ETH0 I1122 00:36:26.442696 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:36:26.442710 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:36:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:36:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:36:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:36:36.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:36:36.412876 3798194 memory.go:184] no items to output this cycle I1122 00:36:36.413019 3798194 cpu.go:275] no items to output this cycle E1122 00:36:46.413628 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:36:46.413655 3798194 memory.go:184] no items to output this cycle I1122 00:36:46.413801 3798194 cpu.go:275] no items to output this cycle E1122 00:36:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:36:56.413265 3798194 memory.go:191] Add success. W1122 00:36:56.413292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:36:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:36:56.413307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:36:56.417711 3798194 cpu.go:282] Add success. I1122 00:36:56.441089 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:36:56.441578 3798194 net.go:642] Add success. I1122 00:36:56.443265 3798194 net.go:778] primary dev: ETH0 I1122 00:36:56.443279 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:36:56.443291 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:36:57.450511 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:36:57.450674 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:36:57.450727 3798194 disk_worker.go:723] disk space is not compliant W1122 00:36:57.450730 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:36:57.459064 3798194 disk_worker.go:509] system disk:vda1 I1122 00:36:57.459108 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:36:58.446375 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:36:59.450033 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:36:59.450129 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:36:59.450157 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:36:59.471669 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:37:00.762163 3798194 disk_info.go:125] begin check local disk info of client I1122 00:37:00.766014 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:37:00.766024 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000468d80 0xc000468dc0] E1122 00:37:06.413302 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:37:06.413326 3798194 memory.go:184] no items to output this cycle I1122 00:37:06.413469 3798194 cpu.go:275] no items to output this cycle E1122 00:37:16.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:37:16.413271 3798194 memory.go:184] no items to output this cycle I1122 00:37:16.413423 3798194 cpu.go:275] no items to output this cycle E1122 00:37:26.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:37:26.412275 3798194 memory.go:191] Add success. I1122 00:37:26.416928 3798194 cpu.go:282] Add success. I1122 00:37:26.441455 3798194 net.go:642] Add success. I1122 00:37:26.442981 3798194 net.go:778] primary dev: ETH0 I1122 00:37:26.442996 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:37:26.443010 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:37:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:37:29.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:37:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:37:36.413049 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:37:36.413074 3798194 memory.go:184] no items to output this cycle I1122 00:37:36.413235 3798194 cpu.go:275] no items to output this cycle E1122 00:37:46.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:37:46.413255 3798194 memory.go:184] no items to output this cycle I1122 00:37:46.413404 3798194 cpu.go:275] no items to output this cycle E1122 00:37:56.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:37:56.413277 3798194 memory.go:191] Add success. W1122 00:37:56.413307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:37:56.413328 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:37:56.413332 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:37:56.417752 3798194 cpu.go:282] Add success. I1122 00:37:56.441161 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:37:56.441731 3798194 net.go:642] Add success. I1122 00:37:56.443300 3798194 net.go:778] primary dev: ETH0 I1122 00:37:56.443324 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:37:56.443337 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:37:57.450635 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:37:57.450803 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:37:57.450857 3798194 disk_worker.go:723] disk space is not compliant W1122 00:37:57.450860 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:37:57.459190 3798194 disk_worker.go:509] system disk:vda1 I1122 00:37:57.459232 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:37:58.446385 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:37:59.450482 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:37:59.450562 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:37:59.450591 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:37:59.471083 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:38:00.766125 3798194 disk_info.go:125] begin check local disk info of client I1122 00:38:00.769946 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:38:00.769955 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037c2c0 0xc00037c300] E1122 00:38:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:38:06.412256 3798194 memory.go:184] no items to output this cycle I1122 00:38:06.412402 3798194 cpu.go:275] no items to output this cycle E1122 00:38:16.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:38:16.413257 3798194 memory.go:184] no items to output this cycle I1122 00:38:16.413399 3798194 cpu.go:275] no items to output this cycle E1122 00:38:26.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:38:26.412260 3798194 memory.go:191] Add success. I1122 00:38:26.416829 3798194 cpu.go:282] Add success. I1122 00:38:26.441274 3798194 net.go:642] Add success. I1122 00:38:26.443013 3798194 net.go:778] primary dev: ETH0 I1122 00:38:26.443027 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:38:26.443040 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:38:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:38:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:38:29.450144 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:38:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:38:36.413257 3798194 memory.go:184] no items to output this cycle I1122 00:38:36.413403 3798194 cpu.go:275] no items to output this cycle E1122 00:38:46.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:38:46.413267 3798194 memory.go:184] no items to output this cycle I1122 00:38:46.413443 3798194 cpu.go:275] no items to output this cycle E1122 00:38:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:38:56.412268 3798194 memory.go:191] Add success. W1122 00:38:56.412304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:38:56.412316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:38:56.412319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:38:56.416559 3798194 cpu.go:282] Add success. I1122 00:38:56.440910 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:38:56.441441 3798194 net.go:642] Add success. I1122 00:38:56.443044 3798194 net.go:778] primary dev: ETH0 I1122 00:38:56.443059 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:38:56.443071 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:38:57.445916 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:38:57.446145 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:38:57.446215 3798194 disk_worker.go:723] disk space is not compliant W1122 00:38:57.446219 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:38:57.448350 3798194 disk_worker.go:509] system disk:vda1 I1122 00:38:57.448389 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:38:58.445985 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:38:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:38:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:38:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:38:59.471644 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:39:00.770060 3798194 disk_info.go:125] begin check local disk info of client I1122 00:39:00.774035 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:39:00.774045 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002f2800 0xc0002f2840] E1122 00:39:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:39:06.412257 3798194 memory.go:184] no items to output this cycle I1122 00:39:06.412396 3798194 cpu.go:275] no items to output this cycle E1122 00:39:16.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:39:16.413268 3798194 memory.go:184] no items to output this cycle I1122 00:39:16.413419 3798194 cpu.go:275] no items to output this cycle I1122 00:39:17.582720 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:39:17.582728 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:39:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:39:26.412271 3798194 memory.go:191] Add success. I1122 00:39:26.416702 3798194 cpu.go:282] Add success. I1122 00:39:26.442242 3798194 net.go:642] Add success. I1122 00:39:26.443797 3798194 net.go:778] primary dev: ETH0 I1122 00:39:26.443811 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:39:26.443826 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:39:29.450470 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:39:29.450547 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:39:29.450571 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:39:36.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:39:36.413260 3798194 memory.go:184] no items to output this cycle I1122 00:39:36.413414 3798194 cpu.go:275] no items to output this cycle E1122 00:39:46.411526 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:39:46.411548 3798194 memory.go:184] no items to output this cycle I1122 00:39:46.411693 3798194 cpu.go:275] no items to output this cycle E1122 00:39:56.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:39:56.412281 3798194 memory.go:191] Add success. W1122 00:39:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:39:56.412323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:39:56.412326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:39:56.417397 3798194 cpu.go:282] Add success. I1122 00:39:56.441887 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:39:56.442452 3798194 net.go:642] Add success. I1122 00:39:56.444105 3798194 net.go:778] primary dev: ETH0 I1122 00:39:56.444134 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:39:56.444148 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:39:57.450622 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:39:57.450778 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:39:57.450828 3798194 disk_worker.go:723] disk space is not compliant W1122 00:39:57.450831 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:39:57.459241 3798194 disk_worker.go:509] system disk:vda1 I1122 00:39:57.459281 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:39:58.445926 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:39:59.450030 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:39:59.450119 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:39:59.450149 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:39:59.471653 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:40:00.774157 3798194 disk_info.go:125] begin check local disk info of client I1122 00:40:00.777977 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:40:00.777985 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0140 0xc0002e0180] E1122 00:40:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:40:06.411977 3798194 memory.go:184] no items to output this cycle I1122 00:40:06.412134 3798194 cpu.go:275] no items to output this cycle E1122 00:40:16.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:40:16.412267 3798194 memory.go:184] no items to output this cycle I1122 00:40:16.412417 3798194 cpu.go:275] no items to output this cycle E1122 00:40:26.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:40:26.412271 3798194 memory.go:191] Add success. I1122 00:40:26.416655 3798194 cpu.go:282] Add success. I1122 00:40:26.442185 3798194 net.go:642] Add success. I1122 00:40:26.443662 3798194 net.go:778] primary dev: ETH0 I1122 00:40:26.443677 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:40:26.443689 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:40:29.450614 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:40:29.450690 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:40:29.450720 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:40:36.412152 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:40:36.412175 3798194 memory.go:184] no items to output this cycle I1122 00:40:36.412319 3798194 cpu.go:275] no items to output this cycle E1122 00:40:46.412472 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:40:46.412493 3798194 memory.go:184] no items to output this cycle I1122 00:40:46.412638 3798194 cpu.go:275] no items to output this cycle E1122 00:40:56.412253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:40:56.412286 3798194 memory.go:191] Add success. W1122 00:40:56.412316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:40:56.412330 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:40:56.412333 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:40:56.416762 3798194 cpu.go:282] Add success. I1122 00:40:56.444033 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:40:56.444538 3798194 net.go:642] Add success. I1122 00:40:56.446138 3798194 net.go:778] primary dev: ETH0 I1122 00:40:56.446154 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:40:56.446167 3798194 net.go:659] Add eni metrics success, count: 7 E1122 00:40:57.450381 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 00:40:57.450395 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:40:57.450555 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:40:57.450609 3798194 disk_worker.go:723] disk space is not compliant W1122 00:40:57.450612 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:40:57.459711 3798194 disk_worker.go:509] system disk:vda1 I1122 00:40:57.459751 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 00:40:58.447730 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 00:40:58.447757 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 00:40:59.448855 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 00:40:59.449939 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:40:59.449993 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:40:59.450012 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:40:59.471547 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:41:00.778091 3798194 disk_info.go:125] begin check local disk info of client I1122 00:41:00.781920 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:41:00.781929 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ff940 0xc0003ff980] E1122 00:41:06.412308 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:41:06.412330 3798194 memory.go:184] no items to output this cycle I1122 00:41:06.412476 3798194 cpu.go:275] no items to output this cycle E1122 00:41:16.412679 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:41:16.412703 3798194 memory.go:184] no items to output this cycle I1122 00:41:16.412849 3798194 cpu.go:275] no items to output this cycle E1122 00:41:26.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:41:26.412902 3798194 memory.go:191] Add success. I1122 00:41:26.417052 3798194 cpu.go:282] Add success. I1122 00:41:26.441576 3798194 net.go:642] Add success. I1122 00:41:26.442970 3798194 net.go:778] primary dev: ETH0 I1122 00:41:26.442985 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:41:26.442999 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:41:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:41:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:41:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:41:36.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:41:36.412889 3798194 memory.go:184] no items to output this cycle I1122 00:41:36.413037 3798194 cpu.go:275] no items to output this cycle E1122 00:41:46.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:41:46.412260 3798194 memory.go:184] no items to output this cycle I1122 00:41:46.412408 3798194 cpu.go:275] no items to output this cycle E1122 00:41:56.412846 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:41:56.412887 3798194 memory.go:191] Add success. W1122 00:41:56.412920 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:41:56.412932 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:41:56.412935 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:41:56.417229 3798194 cpu.go:282] Add success. I1122 00:41:56.441613 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:41:56.442083 3798194 net.go:642] Add success. I1122 00:41:56.443792 3798194 net.go:778] primary dev: ETH0 I1122 00:41:56.443807 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:41:56.443818 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:41:57.450240 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:41:57.450438 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:41:57.450448 3798194 disk_worker.go:723] disk space is not compliant W1122 00:41:57.450451 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:41:57.458441 3798194 disk_worker.go:509] system disk:vda1 I1122 00:41:57.458481 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:41:58.446375 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:41:59.450408 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:41:59.450494 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:41:59.450519 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:41:59.471035 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:42:00.782025 3798194 disk_info.go:125] begin check local disk info of client I1122 00:42:00.785868 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:42:00.785878 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e83c0 0xc0003e8400] E1122 00:42:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:42:06.413255 3798194 memory.go:184] no items to output this cycle I1122 00:42:06.413396 3798194 cpu.go:275] no items to output this cycle E1122 00:42:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:42:16.412261 3798194 memory.go:184] no items to output this cycle I1122 00:42:16.412411 3798194 cpu.go:275] no items to output this cycle I1122 00:42:17.583418 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:42:17.583426 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:42:26.413056 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:42:26.413089 3798194 memory.go:191] Add success. I1122 00:42:26.417831 3798194 cpu.go:282] Add success. I1122 00:42:26.442208 3798194 net.go:642] Add success. I1122 00:42:26.443710 3798194 net.go:778] primary dev: ETH0 I1122 00:42:26.443725 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:42:26.443738 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:42:29.450615 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:42:29.450699 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:42:29.450736 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:42:36.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:42:36.413253 3798194 memory.go:184] no items to output this cycle I1122 00:42:36.413409 3798194 cpu.go:275] no items to output this cycle E1122 00:42:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:42:46.411970 3798194 memory.go:184] no items to output this cycle I1122 00:42:46.412131 3798194 cpu.go:275] no items to output this cycle E1122 00:42:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:42:56.413265 3798194 memory.go:191] Add success. W1122 00:42:56.413300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:42:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:42:56.413315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:42:56.417331 3798194 cpu.go:282] Add success. I1122 00:42:56.441696 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:42:56.442210 3798194 net.go:642] Add success. I1122 00:42:56.443837 3798194 net.go:778] primary dev: ETH0 I1122 00:42:56.443852 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:42:56.443864 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:42:57.451028 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:42:57.451291 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:42:57.451302 3798194 disk_worker.go:723] disk space is not compliant W1122 00:42:57.451305 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:42:57.460065 3798194 disk_worker.go:509] system disk:vda1 I1122 00:42:57.460122 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:42:58.446373 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:42:59.450386 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:42:59.450463 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:42:59.450489 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:42:59.471002 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:43:00.787145 3798194 disk_info.go:125] begin check local disk info of client I1122 00:43:00.790993 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:43:00.791001 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000448840 0xc000448880] E1122 00:43:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:43:06.412258 3798194 memory.go:184] no items to output this cycle I1122 00:43:06.412402 3798194 cpu.go:275] no items to output this cycle E1122 00:43:16.413413 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:43:16.413436 3798194 memory.go:184] no items to output this cycle I1122 00:43:16.413591 3798194 cpu.go:275] no items to output this cycle E1122 00:43:26.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:43:26.412884 3798194 memory.go:191] Add success. I1122 00:43:26.417141 3798194 cpu.go:282] Add success. I1122 00:43:26.441612 3798194 net.go:642] Add success. I1122 00:43:26.442951 3798194 net.go:778] primary dev: ETH0 I1122 00:43:26.442965 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:43:26.442979 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:43:29.449668 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:43:29.449747 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:43:29.449772 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:43:36.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:43:36.412871 3798194 memory.go:184] no items to output this cycle I1122 00:43:36.413015 3798194 cpu.go:275] no items to output this cycle E1122 00:43:46.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:43:46.413262 3798194 memory.go:184] no items to output this cycle I1122 00:43:46.413406 3798194 cpu.go:275] no items to output this cycle E1122 00:43:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:43:56.412059 3798194 memory.go:191] Add success. W1122 00:43:56.412092 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:43:56.412105 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:43:56.412119 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:43:56.412693 3798194 cpu.go:282] Add success. I1122 00:43:56.440994 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:43:56.441588 3798194 net.go:642] Add success. I1122 00:43:56.445307 3798194 net.go:778] primary dev: ETH0 I1122 00:43:56.445321 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:43:56.445333 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:43:57.451022 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:43:57.451255 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:43:57.451267 3798194 disk_worker.go:723] disk space is not compliant W1122 00:43:57.451270 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:43:57.459954 3798194 disk_worker.go:509] system disk:vda1 I1122 00:43:57.459994 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:43:58.446377 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:43:59.450479 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:43:59.450553 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:43:59.450580 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:43:59.471128 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:44:00.791234 3798194 disk_info.go:125] begin check local disk info of client I1122 00:44:00.795137 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:44:00.795146 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e100 0xc00030e140] E1122 00:44:06.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:44:06.411993 3798194 memory.go:184] no items to output this cycle I1122 00:44:06.412149 3798194 cpu.go:275] no items to output this cycle E1122 00:44:16.413420 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:44:16.413445 3798194 memory.go:184] no items to output this cycle I1122 00:44:16.413600 3798194 cpu.go:275] no items to output this cycle E1122 00:44:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:44:26.411953 3798194 memory.go:191] Add success. I1122 00:44:26.412311 3798194 cpu.go:282] Add success. I1122 00:44:26.441940 3798194 net.go:642] Add success. I1122 00:44:26.453520 3798194 net.go:778] primary dev: ETH0 I1122 00:44:26.453537 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:44:26.453551 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:44:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:44:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:44:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:44:36.412959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:44:36.412981 3798194 memory.go:184] no items to output this cycle I1122 00:44:36.413147 3798194 cpu.go:275] no items to output this cycle E1122 00:44:46.413302 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:44:46.413325 3798194 memory.go:184] no items to output this cycle I1122 00:44:46.413470 3798194 cpu.go:275] no items to output this cycle E1122 00:44:56.412480 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:44:56.412513 3798194 memory.go:191] Add success. W1122 00:44:56.412545 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:44:56.412556 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:44:56.412560 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:44:56.417062 3798194 cpu.go:282] Add success. I1122 00:44:56.441606 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:44:56.442199 3798194 net.go:642] Add success. I1122 00:44:56.443699 3798194 net.go:778] primary dev: ETH0 I1122 00:44:56.443712 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:44:56.443726 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:44:57.451051 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:44:57.451238 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:44:57.451297 3798194 disk_worker.go:723] disk space is not compliant W1122 00:44:57.451301 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:44:57.459725 3798194 disk_worker.go:509] system disk:vda1 I1122 00:44:57.459770 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:44:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:44:59.450476 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:44:59.450561 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:44:59.450589 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:44:59.471121 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:45:00.796940 3798194 disk_info.go:125] begin check local disk info of client I1122 00:45:00.800766 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:45:00.800775 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041c200 0xc00041c240] E1122 00:45:06.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:45:06.413269 3798194 memory.go:184] no items to output this cycle I1122 00:45:06.413425 3798194 cpu.go:275] no items to output this cycle E1122 00:45:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:45:16.411972 3798194 memory.go:184] no items to output this cycle I1122 00:45:16.412134 3798194 cpu.go:275] no items to output this cycle I1122 00:45:17.583585 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:45:17.583592 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:45:26.411230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:45:26.411263 3798194 memory.go:191] Add success. I1122 00:45:26.411595 3798194 cpu.go:282] Add success. I1122 00:45:26.442218 3798194 net.go:642] Add success. I1122 00:45:26.445945 3798194 net.go:778] primary dev: ETH0 I1122 00:45:26.445960 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:45:26.445972 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:45:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:45:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:45:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:45:36.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:45:36.413262 3798194 memory.go:184] no items to output this cycle I1122 00:45:36.413404 3798194 cpu.go:275] no items to output this cycle E1122 00:45:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:45:46.411969 3798194 memory.go:184] no items to output this cycle I1122 00:45:46.412137 3798194 cpu.go:275] no items to output this cycle E1122 00:45:56.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:45:56.412882 3798194 memory.go:191] Add success. W1122 00:45:56.412916 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:45:56.412930 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:45:56.412933 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:45:56.417144 3798194 cpu.go:282] Add success. I1122 00:45:56.441470 3798194 event_worker.go:152] Polling the log file for events... I1122 00:45:56.444568 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:45:56.445045 3798194 net.go:642] Add success. I1122 00:45:56.446468 3798194 net.go:778] primary dev: ETH0 I1122 00:45:56.446484 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:45:56.446495 3798194 net.go:659] Add eni metrics success, count: 7 E1122 00:45:57.450843 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 00:45:57.450857 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:45:57.451056 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:45:57.451065 3798194 disk_worker.go:723] disk space is not compliant W1122 00:45:57.451068 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:45:57.459889 3798194 disk_worker.go:509] system disk:vda1 I1122 00:45:57.459931 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 00:45:58.448581 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 00:45:58.448596 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 00:45:59.448979 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 00:45:59.450055 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:45:59.450117 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:45:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:45:59.471662 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:46:00.801144 3798194 disk_info.go:125] begin check local disk info of client I1122 00:46:00.804982 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:46:00.804990 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a22c0 0xc0003a2300] E1122 00:46:06.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:46:06.413261 3798194 memory.go:184] no items to output this cycle I1122 00:46:06.413402 3798194 cpu.go:275] no items to output this cycle E1122 00:46:16.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:46:16.411991 3798194 memory.go:184] no items to output this cycle I1122 00:46:16.412151 3798194 cpu.go:275] no items to output this cycle E1122 00:46:26.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:46:26.413272 3798194 memory.go:191] Add success. I1122 00:46:26.417399 3798194 cpu.go:282] Add success. I1122 00:46:26.442352 3798194 net.go:642] Add success. I1122 00:46:26.443847 3798194 net.go:778] primary dev: ETH0 I1122 00:46:26.443862 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:46:26.443881 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:46:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:46:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:46:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:46:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:46:36.411978 3798194 memory.go:184] no items to output this cycle I1122 00:46:36.412138 3798194 cpu.go:275] no items to output this cycle E1122 00:46:46.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:46:46.413253 3798194 memory.go:184] no items to output this cycle I1122 00:46:46.413394 3798194 cpu.go:275] no items to output this cycle E1122 00:46:56.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:46:56.413265 3798194 memory.go:191] Add success. W1122 00:46:56.413296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:46:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:46:56.413312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:46:56.417725 3798194 cpu.go:282] Add success. I1122 00:46:56.441271 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:46:56.441738 3798194 net.go:642] Add success. I1122 00:46:56.443434 3798194 net.go:778] primary dev: ETH0 I1122 00:46:56.443449 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:46:56.443461 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:46:57.451358 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:46:57.451523 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:46:57.451577 3798194 disk_worker.go:723] disk space is not compliant W1122 00:46:57.451581 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:46:57.459921 3798194 disk_worker.go:509] system disk:vda1 I1122 00:46:57.459964 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:46:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:46:59.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:46:59.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:46:59.450498 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:46:59.471027 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:47:00.805093 3798194 disk_info.go:125] begin check local disk info of client I1122 00:47:00.808947 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:47:00.808955 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040a700 0xc00040a740] E1122 00:47:06.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:47:06.411988 3798194 memory.go:184] no items to output this cycle I1122 00:47:06.412143 3798194 cpu.go:275] no items to output this cycle E1122 00:47:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:47:16.411976 3798194 memory.go:184] no items to output this cycle I1122 00:47:16.412126 3798194 cpu.go:275] no items to output this cycle E1122 00:47:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:47:26.411945 3798194 memory.go:191] Add success. I1122 00:47:26.412303 3798194 cpu.go:282] Add success. I1122 00:47:26.442216 3798194 net.go:642] Add success. I1122 00:47:26.446318 3798194 net.go:778] primary dev: ETH0 I1122 00:47:26.446334 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:47:26.446347 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:47:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:47:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:47:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:47:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:47:36.413254 3798194 memory.go:184] no items to output this cycle I1122 00:47:36.413395 3798194 cpu.go:275] no items to output this cycle E1122 00:47:46.411332 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:47:46.411354 3798194 memory.go:184] no items to output this cycle I1122 00:47:46.411499 3798194 cpu.go:275] no items to output this cycle E1122 00:47:56.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:47:56.412276 3798194 memory.go:191] Add success. W1122 00:47:56.412301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:47:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:47:56.412316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:47:56.416487 3798194 cpu.go:282] Add success. I1122 00:47:56.442072 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:47:56.442978 3798194 net.go:642] Add success. I1122 00:47:56.444469 3798194 net.go:778] primary dev: ETH0 I1122 00:47:56.444484 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:47:56.444500 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:47:57.450883 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:47:57.451035 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:47:57.451088 3798194 disk_worker.go:723] disk space is not compliant W1122 00:47:57.451091 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:47:57.459856 3798194 disk_worker.go:509] system disk:vda1 I1122 00:47:57.459898 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:47:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:47:59.450032 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:47:59.450137 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:47:59.450166 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:47:59.471693 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:48:00.809064 3798194 disk_info.go:125] begin check local disk info of client I1122 00:48:00.812941 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:48:00.812950 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004ae280 0xc0004ae2c0] E1122 00:48:06.413249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:48:06.413271 3798194 memory.go:184] no items to output this cycle I1122 00:48:06.413420 3798194 cpu.go:275] no items to output this cycle E1122 00:48:16.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:48:16.413261 3798194 memory.go:184] no items to output this cycle I1122 00:48:16.413431 3798194 cpu.go:275] no items to output this cycle I1122 00:48:17.583748 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:48:17.583756 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:48:26.412867 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:48:26.412904 3798194 memory.go:191] Add success. I1122 00:48:26.417217 3798194 cpu.go:282] Add success. I1122 00:48:26.441687 3798194 net.go:642] Add success. I1122 00:48:26.443163 3798194 net.go:778] primary dev: ETH0 I1122 00:48:26.443177 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:48:26.443190 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:48:29.450131 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:48:29.450212 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:48:29.450237 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:48:36.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:48:36.412263 3798194 memory.go:184] no items to output this cycle I1122 00:48:36.412406 3798194 cpu.go:275] no items to output this cycle E1122 00:48:46.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:48:46.412252 3798194 memory.go:184] no items to output this cycle I1122 00:48:46.412395 3798194 cpu.go:275] no items to output this cycle E1122 00:48:56.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:48:56.412264 3798194 memory.go:191] Add success. W1122 00:48:56.412296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:48:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:48:56.412311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:48:56.416461 3798194 cpu.go:282] Add success. I1122 00:48:56.441837 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:48:56.442334 3798194 net.go:642] Add success. I1122 00:48:56.443909 3798194 net.go:778] primary dev: ETH0 I1122 00:48:56.443923 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:48:56.443935 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:48:57.450933 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:48:57.451173 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:48:57.451185 3798194 disk_worker.go:723] disk space is not compliant W1122 00:48:57.451188 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:48:57.459979 3798194 disk_worker.go:509] system disk:vda1 I1122 00:48:57.460021 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:48:58.446383 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:48:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:48:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:48:59.450140 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:48:59.471675 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:49:00.813041 3798194 disk_info.go:125] begin check local disk info of client I1122 00:49:00.816929 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:49:00.816938 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000370900 0xc000370940] E1122 00:49:06.412865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:49:06.412886 3798194 memory.go:184] no items to output this cycle I1122 00:49:06.413227 3798194 cpu.go:275] no items to output this cycle E1122 00:49:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:49:16.412263 3798194 memory.go:184] no items to output this cycle I1122 00:49:16.412410 3798194 cpu.go:275] no items to output this cycle E1122 00:49:26.413016 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:49:26.413053 3798194 memory.go:191] Add success. I1122 00:49:26.419952 3798194 cpu.go:282] Add success. I1122 00:49:26.441586 3798194 net.go:642] Add success. I1122 00:49:26.443221 3798194 net.go:778] primary dev: ETH0 I1122 00:49:26.443237 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:49:26.443251 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:49:29.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:49:29.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:49:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:49:36.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:49:36.412878 3798194 memory.go:184] no items to output this cycle I1122 00:49:36.413035 3798194 cpu.go:275] no items to output this cycle E1122 00:49:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:49:46.413259 3798194 memory.go:184] no items to output this cycle I1122 00:49:46.413417 3798194 cpu.go:275] no items to output this cycle E1122 00:49:56.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:49:56.413261 3798194 memory.go:191] Add success. W1122 00:49:56.413291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:49:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:49:56.413307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:49:56.417795 3798194 cpu.go:282] Add success. I1122 00:49:56.441358 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:49:56.441840 3798194 net.go:642] Add success. I1122 00:49:56.443388 3798194 net.go:778] primary dev: ETH0 I1122 00:49:56.443403 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:49:56.443416 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:49:57.450970 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:49:57.451202 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:49:57.451214 3798194 disk_worker.go:723] disk space is not compliant W1122 00:49:57.451217 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:49:57.459957 3798194 disk_worker.go:509] system disk:vda1 I1122 00:49:57.459999 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:49:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:49:59.450408 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:49:59.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:49:59.450510 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:49:59.471028 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:50:00.818147 3798194 disk_info.go:125] begin check local disk info of client I1122 00:50:00.822002 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:50:00.822013 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036ef80 0xc00036efc0] E1122 00:50:06.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:50:06.413271 3798194 memory.go:184] no items to output this cycle I1122 00:50:06.413409 3798194 cpu.go:275] no items to output this cycle E1122 00:50:16.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:50:16.412254 3798194 memory.go:184] no items to output this cycle I1122 00:50:16.412398 3798194 cpu.go:275] no items to output this cycle E1122 00:50:26.412408 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:50:26.412443 3798194 memory.go:191] Add success. I1122 00:50:26.416668 3798194 cpu.go:282] Add success. I1122 00:50:26.442213 3798194 net.go:642] Add success. I1122 00:50:26.443696 3798194 net.go:778] primary dev: ETH0 I1122 00:50:26.443712 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:50:26.443724 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:50:29.449818 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:50:29.449900 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:50:29.449924 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:50:36.412469 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:50:36.412492 3798194 memory.go:184] no items to output this cycle I1122 00:50:36.412634 3798194 cpu.go:275] no items to output this cycle E1122 00:50:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:50:46.413259 3798194 memory.go:184] no items to output this cycle I1122 00:50:46.413404 3798194 cpu.go:275] no items to output this cycle E1122 00:50:56.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:50:56.412267 3798194 memory.go:191] Add success. W1122 00:50:56.412294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:50:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:50:56.412310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:50:56.416764 3798194 cpu.go:282] Add success. I1122 00:50:56.443991 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:50:56.444555 3798194 net.go:642] Add success. I1122 00:50:56.446147 3798194 net.go:778] primary dev: ETH0 I1122 00:50:56.446161 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:50:56.446176 3798194 net.go:659] Add eni metrics success, count: 7 E1122 00:50:57.451097 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 00:50:57.451134 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:50:57.451339 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:50:57.451349 3798194 disk_worker.go:723] disk space is not compliant W1122 00:50:57.451352 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:50:57.460070 3798194 disk_worker.go:509] system disk:vda1 I1122 00:50:57.460126 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 00:50:58.448717 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 00:50:58.448730 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 00:50:59.448618 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 00:50:59.449693 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:50:59.449746 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:50:59.449768 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:50:59.471273 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:51:00.823474 3798194 disk_info.go:125] begin check local disk info of client I1122 00:51:00.827320 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:51:00.827329 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000441040 0xc000441080] E1122 00:51:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:51:06.411981 3798194 memory.go:184] no items to output this cycle I1122 00:51:06.412142 3798194 cpu.go:275] no items to output this cycle E1122 00:51:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:51:16.411976 3798194 memory.go:184] no items to output this cycle I1122 00:51:16.412143 3798194 cpu.go:275] no items to output this cycle I1122 00:51:17.584209 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:51:17.584217 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:51:26.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:51:26.413271 3798194 memory.go:191] Add success. I1122 00:51:26.417378 3798194 cpu.go:282] Add success. I1122 00:51:26.441888 3798194 net.go:642] Add success. I1122 00:51:26.443374 3798194 net.go:778] primary dev: ETH0 I1122 00:51:26.443388 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:51:26.443402 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:51:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:51:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:51:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:51:36.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:51:36.411985 3798194 memory.go:184] no items to output this cycle I1122 00:51:36.412142 3798194 cpu.go:275] no items to output this cycle E1122 00:51:46.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:51:46.412263 3798194 memory.go:184] no items to output this cycle I1122 00:51:46.412408 3798194 cpu.go:275] no items to output this cycle E1122 00:51:56.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:51:56.413270 3798194 memory.go:191] Add success. W1122 00:51:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:51:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:51:56.413313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:51:56.417484 3798194 cpu.go:282] Add success. I1122 00:51:56.441878 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:51:56.442628 3798194 net.go:642] Add success. I1122 00:51:56.444398 3798194 net.go:778] primary dev: ETH0 I1122 00:51:56.444413 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:51:56.444433 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:51:57.451307 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:51:57.451466 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:51:57.451517 3798194 disk_worker.go:723] disk space is not compliant W1122 00:51:57.451520 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:51:57.459887 3798194 disk_worker.go:509] system disk:vda1 I1122 00:51:57.459929 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:51:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:51:59.450485 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:51:59.450566 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:51:59.450594 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:51:59.471131 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:52:00.829145 3798194 disk_info.go:125] begin check local disk info of client I1122 00:52:00.833025 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:52:00.833034 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004baa80 0xc0004baac0] E1122 00:52:06.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:52:06.413263 3798194 memory.go:184] no items to output this cycle I1122 00:52:06.413407 3798194 cpu.go:275] no items to output this cycle E1122 00:52:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:52:16.412254 3798194 memory.go:184] no items to output this cycle I1122 00:52:16.412408 3798194 cpu.go:275] no items to output this cycle E1122 00:52:26.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:52:26.412270 3798194 memory.go:191] Add success. I1122 00:52:26.416492 3798194 cpu.go:282] Add success. I1122 00:52:26.442062 3798194 net.go:642] Add success. I1122 00:52:26.443702 3798194 net.go:778] primary dev: ETH0 I1122 00:52:26.443716 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:52:26.443729 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:52:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:52:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:52:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:52:36.412728 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:52:36.412751 3798194 memory.go:184] no items to output this cycle I1122 00:52:36.412893 3798194 cpu.go:275] no items to output this cycle E1122 00:52:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:52:46.413255 3798194 memory.go:184] no items to output this cycle I1122 00:52:46.413404 3798194 cpu.go:275] no items to output this cycle E1122 00:52:56.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:52:56.412271 3798194 memory.go:191] Add success. W1122 00:52:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:52:56.412317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:52:56.412321 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:52:56.416703 3798194 cpu.go:282] Add success. I1122 00:52:56.441244 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:52:56.441723 3798194 net.go:642] Add success. I1122 00:52:56.443204 3798194 net.go:778] primary dev: ETH0 I1122 00:52:56.443218 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:52:56.443230 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:52:57.450274 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:52:57.450481 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:52:57.450492 3798194 disk_worker.go:723] disk space is not compliant W1122 00:52:57.450495 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:52:57.460250 3798194 disk_worker.go:509] system disk:vda1 I1122 00:52:57.460293 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:52:58.446400 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:52:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:52:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:52:59.450127 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:52:59.471642 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:53:00.838139 3798194 disk_info.go:125] begin check local disk info of client I1122 00:53:00.858714 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:53:00.858725 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fc880 0xc0003fc8c0] E1122 00:53:06.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:53:06.411989 3798194 memory.go:184] no items to output this cycle I1122 00:53:06.412159 3798194 cpu.go:275] no items to output this cycle E1122 00:53:16.412016 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:53:16.412059 3798194 memory.go:184] no items to output this cycle I1122 00:53:16.412328 3798194 cpu.go:275] no items to output this cycle E1122 00:53:26.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:53:26.411952 3798194 memory.go:191] Add success. I1122 00:53:26.412339 3798194 cpu.go:282] Add success. I1122 00:53:26.441995 3798194 net.go:642] Add success. I1122 00:53:26.445805 3798194 net.go:778] primary dev: ETH0 I1122 00:53:26.445826 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:53:26.445841 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:53:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:53:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:53:29.450147 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:53:36.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:53:36.412257 3798194 memory.go:184] no items to output this cycle I1122 00:53:36.412403 3798194 cpu.go:275] no items to output this cycle E1122 00:53:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:53:46.411979 3798194 memory.go:184] no items to output this cycle I1122 00:53:46.412142 3798194 cpu.go:275] no items to output this cycle E1122 00:53:56.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:53:56.412887 3798194 memory.go:191] Add success. W1122 00:53:56.412926 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:53:56.412938 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:53:56.412942 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:53:56.417504 3798194 cpu.go:282] Add success. I1122 00:53:56.441895 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:53:56.442450 3798194 net.go:642] Add success. I1122 00:53:56.444044 3798194 net.go:778] primary dev: ETH0 I1122 00:53:56.444058 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:53:56.444072 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:53:57.451897 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:53:57.452073 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:53:57.452151 3798194 disk_worker.go:723] disk space is not compliant W1122 00:53:57.452154 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:53:57.463630 3798194 disk_worker.go:509] system disk:vda1 I1122 00:53:57.463670 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:53:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:53:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:53:59.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:53:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:53:59.471679 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:54:00.866151 3798194 disk_info.go:125] begin check local disk info of client I1122 00:54:00.883459 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:54:00.883470 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000344340 0xc000344380] E1122 00:54:06.411976 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:54:06.411999 3798194 memory.go:184] no items to output this cycle I1122 00:54:06.412165 3798194 cpu.go:275] no items to output this cycle E1122 00:54:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:54:16.411977 3798194 memory.go:184] no items to output this cycle I1122 00:54:16.412145 3798194 cpu.go:275] no items to output this cycle I1122 00:54:17.584388 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:54:17.584396 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:54:26.411479 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:54:26.411511 3798194 memory.go:191] Add success. I1122 00:54:26.411839 3798194 cpu.go:282] Add success. I1122 00:54:26.441534 3798194 net.go:642] Add success. I1122 00:54:26.445675 3798194 net.go:778] primary dev: ETH0 I1122 00:54:26.445690 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:54:26.445705 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:54:29.450466 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:54:29.450547 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:54:29.450582 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:54:36.412944 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:54:36.412968 3798194 memory.go:184] no items to output this cycle I1122 00:54:36.413126 3798194 cpu.go:275] no items to output this cycle E1122 00:54:46.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:54:46.411990 3798194 memory.go:184] no items to output this cycle I1122 00:54:46.412161 3798194 cpu.go:275] no items to output this cycle E1122 00:54:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:54:56.411959 3798194 memory.go:191] Add success. W1122 00:54:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:54:56.412005 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:54:56.412008 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:54:56.412375 3798194 cpu.go:282] Add success. I1122 00:54:56.441829 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:54:56.442430 3798194 net.go:642] Add success. I1122 00:54:56.446647 3798194 net.go:778] primary dev: ETH0 I1122 00:54:56.446665 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:54:56.446684 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:54:57.451447 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:54:57.451664 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:54:57.451675 3798194 disk_worker.go:723] disk space is not compliant W1122 00:54:57.451678 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:54:57.460726 3798194 disk_worker.go:509] system disk:vda1 I1122 00:54:57.460767 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:54:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:54:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:54:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:54:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:54:59.471666 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:55:00.894216 3798194 disk_info.go:125] begin check local disk info of client I1122 00:55:00.899518 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:55:00.899527 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003444c0 0xc000344500] E1122 00:55:06.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:55:06.413258 3798194 memory.go:184] no items to output this cycle I1122 00:55:06.413407 3798194 cpu.go:275] no items to output this cycle E1122 00:55:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:55:16.411982 3798194 memory.go:184] no items to output this cycle I1122 00:55:16.412146 3798194 cpu.go:275] no items to output this cycle E1122 00:55:26.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:55:26.413283 3798194 memory.go:191] Add success. I1122 00:55:26.417548 3798194 cpu.go:282] Add success. I1122 00:55:26.442272 3798194 net.go:642] Add success. I1122 00:55:26.443757 3798194 net.go:778] primary dev: ETH0 I1122 00:55:26.443771 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:55:26.443785 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:55:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:55:29.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:55:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:55:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:55:36.412252 3798194 memory.go:184] no items to output this cycle I1122 00:55:36.412402 3798194 cpu.go:275] no items to output this cycle E1122 00:55:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:55:46.411974 3798194 memory.go:184] no items to output this cycle I1122 00:55:46.412135 3798194 cpu.go:275] no items to output this cycle E1122 00:55:56.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:55:56.411957 3798194 memory.go:191] Add success. W1122 00:55:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:55:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:55:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:55:56.412349 3798194 cpu.go:282] Add success. I1122 00:55:56.443827 3798194 event_worker.go:152] Polling the log file for events... I1122 00:55:56.444041 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:55:56.444551 3798194 net.go:642] Add success. I1122 00:55:56.448780 3798194 net.go:778] primary dev: ETH0 I1122 00:55:56.448795 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:55:56.448807 3798194 net.go:659] Add eni metrics success, count: 7 E1122 00:55:57.451307 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 00:55:57.451323 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:55:57.451700 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:55:57.451715 3798194 disk_worker.go:723] disk space is not compliant W1122 00:55:57.451718 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:55:57.461138 3798194 disk_worker.go:509] system disk:vda1 I1122 00:55:57.461179 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 00:55:58.447690 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 00:55:58.447704 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 00:55:59.448627 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 00:55:59.449701 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:55:59.449750 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:55:59.449769 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:55:59.471278 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:56:00.901144 3798194 disk_info.go:125] begin check local disk info of client I1122 00:56:00.904989 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:56:00.904998 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000489940 0xc000489980] E1122 00:56:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:56:06.411985 3798194 memory.go:184] no items to output this cycle I1122 00:56:06.412140 3798194 cpu.go:275] no items to output this cycle E1122 00:56:16.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:56:16.411989 3798194 memory.go:184] no items to output this cycle I1122 00:56:16.412147 3798194 cpu.go:275] no items to output this cycle E1122 00:56:26.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:56:26.412269 3798194 memory.go:191] Add success. I1122 00:56:26.416730 3798194 cpu.go:282] Add success. I1122 00:56:26.441421 3798194 net.go:642] Add success. I1122 00:56:26.442906 3798194 net.go:778] primary dev: ETH0 I1122 00:56:26.442921 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:56:26.442933 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:56:29.450037 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:56:29.450129 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:56:29.450156 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:56:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:56:36.411982 3798194 memory.go:184] no items to output this cycle I1122 00:56:36.412138 3798194 cpu.go:275] no items to output this cycle E1122 00:56:46.413274 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:56:46.413307 3798194 memory.go:184] no items to output this cycle I1122 00:56:46.413464 3798194 cpu.go:275] no items to output this cycle E1122 00:56:56.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:56:56.412275 3798194 memory.go:191] Add success. W1122 00:56:56.412319 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:56:56.412331 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:56:56.412334 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:56:56.416709 3798194 cpu.go:282] Add success. I1122 00:56:56.441338 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:56:56.441810 3798194 net.go:642] Add success. I1122 00:56:56.443296 3798194 net.go:778] primary dev: ETH0 I1122 00:56:56.443312 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:56:56.443329 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:56:57.452065 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:56:57.452328 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:56:57.452339 3798194 disk_worker.go:723] disk space is not compliant W1122 00:56:57.452342 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:56:57.462921 3798194 disk_worker.go:509] system disk:vda1 I1122 00:56:57.462964 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:56:58.446366 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:56:59.450467 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:56:59.450564 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:56:59.450596 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:56:59.471127 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:57:00.906135 3798194 disk_info.go:125] begin check local disk info of client I1122 00:57:00.909725 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:57:00.909734 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b4d80 0xc0003b4dc0] E1122 00:57:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:57:06.411977 3798194 memory.go:184] no items to output this cycle I1122 00:57:06.412131 3798194 cpu.go:275] no items to output this cycle E1122 00:57:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:57:16.411974 3798194 memory.go:184] no items to output this cycle I1122 00:57:16.412141 3798194 cpu.go:275] no items to output this cycle I1122 00:57:17.584576 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 00:57:17.584585 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 00:57:26.412869 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:57:26.412907 3798194 memory.go:191] Add success. I1122 00:57:26.417198 3798194 cpu.go:282] Add success. I1122 00:57:26.441695 3798194 net.go:642] Add success. I1122 00:57:26.443471 3798194 net.go:778] primary dev: ETH0 I1122 00:57:26.443485 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:57:26.443505 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:57:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:57:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:57:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:57:36.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:57:36.411982 3798194 memory.go:184] no items to output this cycle I1122 00:57:36.412139 3798194 cpu.go:275] no items to output this cycle E1122 00:57:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:57:46.411968 3798194 memory.go:184] no items to output this cycle I1122 00:57:46.412120 3798194 cpu.go:275] no items to output this cycle E1122 00:57:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:57:56.411948 3798194 memory.go:191] Add success. W1122 00:57:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:57:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:57:56.411990 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:57:56.412333 3798194 cpu.go:282] Add success. I1122 00:57:56.441801 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:57:56.442307 3798194 net.go:642] Add success. I1122 00:57:56.446040 3798194 net.go:778] primary dev: ETH0 I1122 00:57:56.446054 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:57:56.446065 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:57:57.451050 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:57:57.451232 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:57:57.451285 3798194 disk_worker.go:723] disk space is not compliant W1122 00:57:57.451288 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:57:57.460091 3798194 disk_worker.go:509] system disk:vda1 I1122 00:57:57.460148 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:57:58.445650 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:57:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:57:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:57:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:57:59.471637 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:58:00.917151 3798194 disk_info.go:125] begin check local disk info of client I1122 00:58:00.937549 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:58:00.937559 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00027d400 0xc00027d440] E1122 00:58:06.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:58:06.413262 3798194 memory.go:184] no items to output this cycle I1122 00:58:06.413403 3798194 cpu.go:275] no items to output this cycle E1122 00:58:16.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:58:16.413253 3798194 memory.go:184] no items to output this cycle I1122 00:58:16.413395 3798194 cpu.go:275] no items to output this cycle E1122 00:58:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:58:26.413265 3798194 memory.go:191] Add success. I1122 00:58:26.417504 3798194 cpu.go:282] Add success. I1122 00:58:26.441295 3798194 net.go:642] Add success. I1122 00:58:26.442830 3798194 net.go:778] primary dev: ETH0 I1122 00:58:26.442844 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:58:26.442859 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:58:29.450616 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:58:29.450704 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:58:29.450730 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:58:36.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:58:36.412876 3798194 memory.go:184] no items to output this cycle I1122 00:58:36.413016 3798194 cpu.go:275] no items to output this cycle E1122 00:58:46.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:58:46.413359 3798194 memory.go:184] no items to output this cycle I1122 00:58:46.413688 3798194 cpu.go:275] no items to output this cycle E1122 00:58:56.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:58:56.412275 3798194 memory.go:191] Add success. W1122 00:58:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:58:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:58:56.412316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:58:56.416641 3798194 cpu.go:282] Add success. I1122 00:58:56.441352 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:58:56.441825 3798194 net.go:642] Add success. I1122 00:58:56.443408 3798194 net.go:778] primary dev: ETH0 I1122 00:58:56.443423 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:58:56.443435 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:58:57.450783 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:58:57.450942 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:58:57.451014 3798194 disk_worker.go:723] disk space is not compliant W1122 00:58:57.451018 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:58:57.459181 3798194 disk_worker.go:509] system disk:vda1 I1122 00:58:57.459223 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:58:58.446565 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:58:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:58:59.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:58:59.450495 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:58:59.471001 3798194 disk_local_worker.go:436] Get disk info: [] I1122 00:59:00.937669 3798194 disk_info.go:125] begin check local disk info of client I1122 00:59:00.941550 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 00:59:00.941559 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000266d80 0xc000266dc0] E1122 00:59:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:59:06.412255 3798194 memory.go:184] no items to output this cycle I1122 00:59:06.412400 3798194 cpu.go:275] no items to output this cycle E1122 00:59:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:59:16.412260 3798194 memory.go:184] no items to output this cycle I1122 00:59:16.412404 3798194 cpu.go:275] no items to output this cycle E1122 00:59:26.411982 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:59:26.412015 3798194 memory.go:191] Add success. I1122 00:59:26.416535 3798194 cpu.go:282] Add success. I1122 00:59:26.442239 3798194 net.go:642] Add success. I1122 00:59:26.443868 3798194 net.go:778] primary dev: ETH0 I1122 00:59:26.443883 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:59:26.443894 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:59:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:59:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:59:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 00:59:36.412344 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:59:36.412372 3798194 memory.go:184] no items to output this cycle I1122 00:59:36.412582 3798194 cpu.go:275] no items to output this cycle E1122 00:59:46.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:59:46.412263 3798194 memory.go:184] no items to output this cycle I1122 00:59:46.412408 3798194 cpu.go:275] no items to output this cycle E1122 00:59:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 00:59:56.411947 3798194 memory.go:191] Add success. W1122 00:59:56.411974 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 00:59:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 00:59:56.411989 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 00:59:56.412323 3798194 cpu.go:282] Add success. I1122 00:59:56.441806 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 00:59:56.442326 3798194 net.go:642] Add success. I1122 00:59:56.446614 3798194 net.go:778] primary dev: ETH0 I1122 00:59:56.446629 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 00:59:56.446642 3798194 net.go:659] Add eni metrics success, count: 7 I1122 00:59:57.450621 3798194 custom_config.go:106] query custom config with name: gpu W1122 00:59:57.450924 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 00:59:57.450941 3798194 disk_worker.go:723] disk space is not compliant W1122 00:59:57.450944 3798194 disk_worker.go:743] disk inode is not compliant I1122 00:59:57.459730 3798194 disk_worker.go:509] system disk:vda1 I1122 00:59:57.459769 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 00:59:58.446011 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 00:59:59.450406 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 00:59:59.450490 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 00:59:59.450518 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 00:59:59.471026 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:00:00.941664 3798194 disk_info.go:125] begin check local disk info of client I1122 01:00:00.945181 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:00:00.945190 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00049d700 0xc00049d740] E1122 01:00:06.412977 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:00:06.413009 3798194 memory.go:184] no items to output this cycle I1122 01:00:06.413174 3798194 cpu.go:275] no items to output this cycle E1122 01:00:16.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:00:16.411987 3798194 memory.go:184] no items to output this cycle I1122 01:00:16.412145 3798194 cpu.go:275] no items to output this cycle I1122 01:00:17.584718 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:00:17.584727 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:00:26.411307 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:00:26.411342 3798194 memory.go:191] Add success. I1122 01:00:26.411677 3798194 cpu.go:282] Add success. I1122 01:00:26.441279 3798194 net.go:642] Add success. I1122 01:00:26.445582 3798194 net.go:778] primary dev: ETH0 I1122 01:00:26.445597 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:00:26.445611 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:00:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:00:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:00:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:00:36.412335 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:00:36.412358 3798194 memory.go:184] no items to output this cycle I1122 01:00:36.412499 3798194 cpu.go:275] no items to output this cycle E1122 01:00:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:00:46.411977 3798194 memory.go:184] no items to output this cycle I1122 01:00:46.412138 3798194 cpu.go:275] no items to output this cycle E1122 01:00:56.412838 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:00:56.412873 3798194 memory.go:191] Add success. W1122 01:00:56.412907 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:00:56.412919 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:00:56.412922 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:00:56.418345 3798194 cpu.go:282] Add success. I1122 01:00:56.442212 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:00:56.442682 3798194 net.go:642] Add success. I1122 01:00:56.444643 3798194 net.go:778] primary dev: ETH0 I1122 01:00:56.444657 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:00:56.444669 3798194 net.go:659] Add eni metrics success, count: 7 E1122 01:00:57.445305 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 01:00:57.445331 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:00:57.445601 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:00:57.445612 3798194 disk_worker.go:723] disk space is not compliant W1122 01:00:57.445615 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:00:57.448121 3798194 disk_worker.go:509] system disk:vda1 I1122 01:00:57.448165 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 01:00:58.448181 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 01:00:58.448196 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 01:00:59.448995 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 01:00:59.450069 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:00:59.450129 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:00:59.450154 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:00:59.471657 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:01:00.945284 3798194 disk_info.go:125] begin check local disk info of client I1122 01:01:00.954093 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:01:00.954103 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b8dc0 0xc0003b8e00] E1122 01:01:06.412001 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:01:06.412029 3798194 memory.go:184] no items to output this cycle I1122 01:01:06.412205 3798194 cpu.go:275] no items to output this cycle E1122 01:01:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:01:16.411980 3798194 memory.go:184] no items to output this cycle I1122 01:01:16.412143 3798194 cpu.go:275] no items to output this cycle E1122 01:01:26.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:01:26.411966 3798194 memory.go:191] Add success. I1122 01:01:26.412330 3798194 cpu.go:282] Add success. I1122 01:01:26.442212 3798194 net.go:642] Add success. I1122 01:01:26.446217 3798194 net.go:778] primary dev: ETH0 I1122 01:01:26.446240 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:01:26.446254 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:01:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:01:29.450122 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:01:29.450155 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:01:36.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:01:36.413251 3798194 memory.go:184] no items to output this cycle I1122 01:01:36.413395 3798194 cpu.go:275] no items to output this cycle E1122 01:01:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:01:46.411973 3798194 memory.go:184] no items to output this cycle I1122 01:01:46.412137 3798194 cpu.go:275] no items to output this cycle E1122 01:01:56.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:01:56.411944 3798194 memory.go:191] Add success. W1122 01:01:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:01:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:01:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:01:56.412326 3798194 cpu.go:282] Add success. I1122 01:01:56.441803 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:01:56.442311 3798194 net.go:642] Add success. I1122 01:01:56.446513 3798194 net.go:778] primary dev: ETH0 I1122 01:01:56.446527 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:01:56.446539 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:01:57.450809 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:01:57.451025 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:01:57.451035 3798194 disk_worker.go:723] disk space is not compliant W1122 01:01:57.451039 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:01:57.461883 3798194 disk_worker.go:509] system disk:vda1 I1122 01:01:57.461927 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:01:58.446008 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:01:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:01:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:01:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:01:59.471666 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:02:00.956146 3798194 disk_info.go:125] begin check local disk info of client I1122 01:02:00.960234 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:02:00.960243 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00027f480 0xc00027f4c0] E1122 01:02:06.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:02:06.412269 3798194 memory.go:184] no items to output this cycle I1122 01:02:06.412413 3798194 cpu.go:275] no items to output this cycle E1122 01:02:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:02:16.411985 3798194 memory.go:184] no items to output this cycle I1122 01:02:16.412148 3798194 cpu.go:275] no items to output this cycle E1122 01:02:26.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:02:26.412895 3798194 memory.go:191] Add success. I1122 01:02:26.418364 3798194 cpu.go:282] Add success. I1122 01:02:26.442150 3798194 net.go:642] Add success. I1122 01:02:26.443872 3798194 net.go:778] primary dev: ETH0 I1122 01:02:26.443888 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:02:26.443905 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:02:29.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:02:29.450128 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:02:29.450154 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:02:36.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:02:36.412874 3798194 memory.go:184] no items to output this cycle I1122 01:02:36.413015 3798194 cpu.go:275] no items to output this cycle E1122 01:02:46.413349 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:02:46.413386 3798194 memory.go:184] no items to output this cycle I1122 01:02:46.413669 3798194 cpu.go:275] no items to output this cycle E1122 01:02:56.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:02:56.413283 3798194 memory.go:191] Add success. W1122 01:02:56.413315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:02:56.413326 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:02:56.413330 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:02:56.417558 3798194 cpu.go:282] Add success. I1122 01:02:56.440935 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:02:56.441522 3798194 net.go:642] Add success. I1122 01:02:56.443076 3798194 net.go:778] primary dev: ETH0 I1122 01:02:56.443091 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:02:56.443106 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:02:57.451348 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:02:57.451561 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:02:57.451572 3798194 disk_worker.go:723] disk space is not compliant W1122 01:02:57.451575 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:02:57.459923 3798194 disk_worker.go:509] system disk:vda1 I1122 01:02:57.459965 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:02:58.446428 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:02:59.450530 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:02:59.450609 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:02:59.450635 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:02:59.471133 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:03:00.962146 3798194 disk_info.go:125] begin check local disk info of client I1122 01:03:00.965833 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:03:00.965842 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000350e40 0xc000350e80] E1122 01:03:06.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:03:06.413269 3798194 memory.go:184] no items to output this cycle I1122 01:03:06.413421 3798194 cpu.go:275] no items to output this cycle E1122 01:03:16.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:03:16.412250 3798194 memory.go:184] no items to output this cycle I1122 01:03:16.412395 3798194 cpu.go:275] no items to output this cycle I1122 01:03:17.584888 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:03:17.584896 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:03:26.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:03:26.412262 3798194 memory.go:191] Add success. I1122 01:03:26.416240 3798194 cpu.go:282] Add success. I1122 01:03:26.441669 3798194 net.go:642] Add success. I1122 01:03:26.443087 3798194 net.go:778] primary dev: ETH0 I1122 01:03:26.443121 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:03:26.443133 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:03:29.450385 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:03:29.450465 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:03:29.450491 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:03:36.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:03:36.413250 3798194 memory.go:184] no items to output this cycle I1122 01:03:36.413389 3798194 cpu.go:275] no items to output this cycle E1122 01:03:46.413016 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:03:46.413038 3798194 memory.go:184] no items to output this cycle I1122 01:03:46.413206 3798194 cpu.go:275] no items to output this cycle E1122 01:03:56.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:03:56.413273 3798194 memory.go:191] Add success. W1122 01:03:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:03:56.413326 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:03:56.413329 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:03:56.417668 3798194 cpu.go:282] Add success. I1122 01:03:56.440948 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:03:56.441547 3798194 net.go:642] Add success. I1122 01:03:56.443095 3798194 net.go:778] primary dev: ETH0 I1122 01:03:56.443121 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:03:56.443135 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:03:57.450692 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:03:57.450922 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:03:57.450934 3798194 disk_worker.go:723] disk space is not compliant W1122 01:03:57.450937 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:03:57.459805 3798194 disk_worker.go:509] system disk:vda1 I1122 01:03:57.459848 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:03:58.446372 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:03:59.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:03:59.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:03:59.450507 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:03:59.471011 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:04:00.967147 3798194 disk_info.go:125] begin check local disk info of client I1122 01:04:00.970652 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:04:00.970661 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046f580 0xc00046f5c0] E1122 01:04:06.413258 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:04:06.413286 3798194 memory.go:184] no items to output this cycle I1122 01:04:06.413436 3798194 cpu.go:275] no items to output this cycle E1122 01:04:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:04:16.411980 3798194 memory.go:184] no items to output this cycle I1122 01:04:16.412146 3798194 cpu.go:275] no items to output this cycle E1122 01:04:26.412869 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:04:26.412909 3798194 memory.go:191] Add success. I1122 01:04:26.417269 3798194 cpu.go:282] Add success. I1122 01:04:26.441763 3798194 net.go:642] Add success. I1122 01:04:26.443284 3798194 net.go:778] primary dev: ETH0 I1122 01:04:26.443300 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:04:26.443315 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:04:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:04:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:04:29.450514 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:04:36.412637 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:04:36.412659 3798194 memory.go:184] no items to output this cycle I1122 01:04:36.412809 3798194 cpu.go:275] no items to output this cycle E1122 01:04:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:04:46.411986 3798194 memory.go:184] no items to output this cycle I1122 01:04:46.412157 3798194 cpu.go:275] no items to output this cycle E1122 01:04:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:04:56.411949 3798194 memory.go:191] Add success. W1122 01:04:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:04:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:04:56.412000 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:04:56.412339 3798194 cpu.go:282] Add success. I1122 01:04:56.441813 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:04:56.442325 3798194 net.go:642] Add success. I1122 01:04:56.446872 3798194 net.go:778] primary dev: ETH0 I1122 01:04:56.446888 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:04:56.446900 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:04:57.450393 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:04:57.450569 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:04:57.450623 3798194 disk_worker.go:723] disk space is not compliant W1122 01:04:57.450626 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:04:57.461402 3798194 disk_worker.go:509] system disk:vda1 I1122 01:04:57.461448 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:04:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:04:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:04:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:04:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:04:59.471644 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:05:00.970766 3798194 disk_info.go:125] begin check local disk info of client I1122 01:05:00.974676 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:05:00.974685 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003dabc0 0xc0003dac00] E1122 01:05:06.412778 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:05:06.412801 3798194 memory.go:184] no items to output this cycle I1122 01:05:06.412953 3798194 cpu.go:275] no items to output this cycle E1122 01:05:16.411944 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:05:16.411972 3798194 memory.go:184] no items to output this cycle I1122 01:05:16.412125 3798194 cpu.go:275] no items to output this cycle E1122 01:05:26.412051 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:05:26.412162 3798194 memory.go:191] Add success. I1122 01:05:26.412841 3798194 cpu.go:282] Add success. I1122 01:05:26.441306 3798194 net.go:642] Add success. I1122 01:05:26.445585 3798194 net.go:778] primary dev: ETH0 I1122 01:05:26.445600 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:05:26.445613 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:05:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:05:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:05:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:05:36.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:05:36.412875 3798194 memory.go:184] no items to output this cycle I1122 01:05:36.413018 3798194 cpu.go:275] no items to output this cycle E1122 01:05:46.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:05:46.411984 3798194 memory.go:184] no items to output this cycle I1122 01:05:46.412145 3798194 cpu.go:275] no items to output this cycle E1122 01:05:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:05:56.411952 3798194 memory.go:191] Add success. W1122 01:05:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:05:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:05:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 W1122 01:05:56.420924 3798194 conf_downlod.go:84] cant't download conf: Get "": unsupported protocol scheme "" W1122 01:05:56.420941 3798194 conf_downlod.go:89] use old conf I1122 01:05:56.421138 3798194 cpu.go:282] Add success. I1122 01:05:56.443997 3798194 event_worker.go:152] Polling the log file for events... I1122 01:05:56.444363 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:05:56.444821 3798194 net.go:642] Add success. I1122 01:05:56.448781 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795278956, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 01:05:56.448906 3798194 net.go:778] primary dev: ETH0 I1122 01:05:56.448920 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:05:56.448933 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:05:56.458031 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 01:05:57.445268 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 01:05:57.445285 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:05:57.445492 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:05:57.445568 3798194 disk_worker.go:723] disk space is not compliant W1122 01:05:57.445571 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:05:57.448062 3798194 disk_worker.go:509] system disk:vda1 I1122 01:05:57.448100 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 01:05:58.447806 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 01:05:58.447821 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 01:05:59.448606 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 01:05:59.449657 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:05:59.449706 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:05:59.449731 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:05:59.471191 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:06:00.974791 3798194 disk_info.go:125] begin check local disk info of client I1122 01:06:00.978340 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:06:00.978350 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000473440 0xc000473480] E1122 01:06:06.412417 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:06:06.412438 3798194 memory.go:184] no items to output this cycle I1122 01:06:06.412599 3798194 cpu.go:275] no items to output this cycle E1122 01:06:16.413398 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:06:16.413420 3798194 memory.go:184] no items to output this cycle I1122 01:06:16.413566 3798194 cpu.go:275] no items to output this cycle I1122 01:06:17.585064 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:06:17.585073 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:06:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:06:26.413264 3798194 memory.go:191] Add success. I1122 01:06:26.417723 3798194 cpu.go:282] Add success. I1122 01:06:26.442245 3798194 net.go:642] Add success. I1122 01:06:26.443758 3798194 net.go:778] primary dev: ETH0 I1122 01:06:26.443774 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:06:26.443789 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:06:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:06:29.450465 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:06:29.450490 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:06:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:06:36.412256 3798194 memory.go:184] no items to output this cycle I1122 01:06:36.412397 3798194 cpu.go:275] no items to output this cycle E1122 01:06:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:06:46.411976 3798194 memory.go:184] no items to output this cycle I1122 01:06:46.412364 3798194 cpu.go:275] no items to output this cycle E1122 01:06:56.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:06:56.413260 3798194 memory.go:191] Add success. W1122 01:06:56.413293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:06:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:06:56.413308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:06:56.417807 3798194 cpu.go:282] Add success. I1122 01:06:56.441219 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:06:56.441699 3798194 net.go:642] Add success. I1122 01:06:56.443297 3798194 net.go:778] primary dev: ETH0 I1122 01:06:56.443312 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:06:56.443324 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:06:57.450702 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:06:57.450866 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:06:57.450921 3798194 disk_worker.go:723] disk space is not compliant W1122 01:06:57.450924 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:06:57.458972 3798194 disk_worker.go:509] system disk:vda1 I1122 01:06:57.459013 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:06:58.446076 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:06:59.450132 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:06:59.450205 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:06:59.450230 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:06:59.470714 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:07:00.980996 3798194 disk_info.go:125] begin check local disk info of client I1122 01:07:00.985180 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:07:00.985190 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00035ea00 0xc00035ea40] E1122 01:07:06.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:07:06.413258 3798194 memory.go:184] no items to output this cycle I1122 01:07:06.413411 3798194 cpu.go:275] no items to output this cycle E1122 01:07:16.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:07:16.412254 3798194 memory.go:184] no items to output this cycle I1122 01:07:16.412401 3798194 cpu.go:275] no items to output this cycle E1122 01:07:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:07:26.411953 3798194 memory.go:191] Add success. I1122 01:07:26.412304 3798194 cpu.go:282] Add success. I1122 01:07:26.441743 3798194 net.go:642] Add success. I1122 01:07:26.446023 3798194 net.go:778] primary dev: ETH0 I1122 01:07:26.446038 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:07:26.446051 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:07:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:07:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:07:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:07:36.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:07:36.412267 3798194 memory.go:184] no items to output this cycle I1122 01:07:36.412410 3798194 cpu.go:275] no items to output this cycle E1122 01:07:46.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:07:46.412878 3798194 memory.go:184] no items to output this cycle I1122 01:07:46.413023 3798194 cpu.go:275] no items to output this cycle E1122 01:07:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:07:56.411948 3798194 memory.go:191] Add success. W1122 01:07:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:07:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:07:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:07:56.412334 3798194 cpu.go:282] Add success. I1122 01:07:56.441810 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:07:56.442340 3798194 net.go:642] Add success. I1122 01:07:56.446622 3798194 net.go:778] primary dev: ETH0 I1122 01:07:56.446636 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:07:56.446649 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:07:57.452296 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:07:57.452505 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:07:57.452518 3798194 disk_worker.go:723] disk space is not compliant W1122 01:07:57.452521 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:07:57.462156 3798194 disk_worker.go:509] system disk:vda1 I1122 01:07:57.462197 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:07:58.446033 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:07:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:07:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:07:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:07:59.471639 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:08:00.987148 3798194 disk_info.go:125] begin check local disk info of client I1122 01:08:00.990957 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:08:00.990966 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000481500 0xc000481540] E1122 01:08:06.412776 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:08:06.412798 3798194 memory.go:184] no items to output this cycle I1122 01:08:06.412955 3798194 cpu.go:275] no items to output this cycle E1122 01:08:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:08:16.411981 3798194 memory.go:184] no items to output this cycle I1122 01:08:16.412135 3798194 cpu.go:275] no items to output this cycle E1122 01:08:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:08:26.411958 3798194 memory.go:191] Add success. I1122 01:08:26.412323 3798194 cpu.go:282] Add success. I1122 01:08:26.441891 3798194 net.go:642] Add success. I1122 01:08:26.445953 3798194 net.go:778] primary dev: ETH0 I1122 01:08:26.445968 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:08:26.445984 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:08:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:08:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:08:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:08:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:08:36.412257 3798194 memory.go:184] no items to output this cycle I1122 01:08:36.412399 3798194 cpu.go:275] no items to output this cycle E1122 01:08:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:08:46.411971 3798194 memory.go:184] no items to output this cycle I1122 01:08:46.412141 3798194 cpu.go:275] no items to output this cycle E1122 01:08:56.411887 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:08:56.411917 3798194 memory.go:191] Add success. W1122 01:08:56.411952 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:08:56.411963 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:08:56.411966 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:08:56.412331 3798194 cpu.go:282] Add success. I1122 01:08:56.441813 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:08:56.442317 3798194 net.go:642] Add success. I1122 01:08:56.446308 3798194 net.go:778] primary dev: ETH0 I1122 01:08:56.446323 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:08:56.446334 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:08:57.447992 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:08:57.448264 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:08:57.448276 3798194 disk_worker.go:723] disk space is not compliant W1122 01:08:57.448280 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:08:57.451035 3798194 disk_worker.go:509] system disk:vda1 I1122 01:08:57.451075 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:08:58.445982 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:08:59.450489 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:08:59.450573 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:08:59.450599 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:08:59.471089 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:09:00.993141 3798194 disk_info.go:125] begin check local disk info of client I1122 01:09:00.997374 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:09:00.997389 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a0bc0 0xc0003a0c00] E1122 01:09:06.412281 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:09:06.412307 3798194 memory.go:184] no items to output this cycle I1122 01:09:06.412460 3798194 cpu.go:275] no items to output this cycle E1122 01:09:16.411943 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:09:16.411966 3798194 memory.go:184] no items to output this cycle I1122 01:09:16.412122 3798194 cpu.go:275] no items to output this cycle I1122 01:09:17.585235 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:09:17.585244 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:09:26.411425 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:09:26.411464 3798194 memory.go:191] Add success. I1122 01:09:26.411834 3798194 cpu.go:282] Add success. I1122 01:09:26.441315 3798194 net.go:642] Add success. I1122 01:09:26.445280 3798194 net.go:778] primary dev: ETH0 I1122 01:09:26.445296 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:09:26.445308 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:09:29.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:09:29.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:09:29.450521 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:09:36.412470 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:09:36.412493 3798194 memory.go:184] no items to output this cycle I1122 01:09:36.412639 3798194 cpu.go:275] no items to output this cycle E1122 01:09:46.412320 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:09:46.412345 3798194 memory.go:184] no items to output this cycle I1122 01:09:46.412493 3798194 cpu.go:275] no items to output this cycle E1122 01:09:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:09:56.412265 3798194 memory.go:191] Add success. W1122 01:09:56.412296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:09:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:09:56.412310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:09:56.417002 3798194 cpu.go:282] Add success. I1122 01:09:56.441364 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:09:56.441836 3798194 net.go:642] Add success. I1122 01:09:56.443641 3798194 net.go:778] primary dev: ETH0 I1122 01:09:56.443656 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:09:56.443669 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:09:57.451062 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:09:57.451296 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:09:57.451307 3798194 disk_worker.go:723] disk space is not compliant W1122 01:09:57.451311 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:09:57.459711 3798194 disk_worker.go:509] system disk:vda1 I1122 01:09:57.459753 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:09:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:09:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:09:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:09:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:09:59.471652 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:10:00.999147 3798194 disk_info.go:125] begin check local disk info of client I1122 01:10:01.002735 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:10:01.002745 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000314c40 0xc000314c80] E1122 01:10:06.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:10:06.413257 3798194 memory.go:184] no items to output this cycle I1122 01:10:06.413407 3798194 cpu.go:275] no items to output this cycle E1122 01:10:16.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:10:16.411988 3798194 memory.go:184] no items to output this cycle I1122 01:10:16.412147 3798194 cpu.go:275] no items to output this cycle E1122 01:10:26.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:10:26.411956 3798194 memory.go:191] Add success. I1122 01:10:26.412301 3798194 cpu.go:282] Add success. I1122 01:10:26.442023 3798194 net.go:642] Add success. I1122 01:10:26.446296 3798194 net.go:778] primary dev: ETH0 I1122 01:10:26.446311 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:10:26.446332 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:10:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:10:29.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:10:29.450149 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:10:36.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:10:36.411976 3798194 memory.go:184] no items to output this cycle I1122 01:10:36.412133 3798194 cpu.go:275] no items to output this cycle E1122 01:10:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:10:46.411982 3798194 memory.go:184] no items to output this cycle I1122 01:10:46.412147 3798194 cpu.go:275] no items to output this cycle E1122 01:10:56.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:10:56.413273 3798194 memory.go:191] Add success. W1122 01:10:56.413307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:10:56.413318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:10:56.413322 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:10:56.418379 3798194 cpu.go:282] Add success. I1122 01:10:56.443002 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:10:56.443521 3798194 net.go:642] Add success. I1122 01:10:56.446600 3798194 net.go:778] primary dev: ETH0 I1122 01:10:56.446614 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:10:56.446627 3798194 net.go:659] Add eni metrics success, count: 7 E1122 01:10:57.447962 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 01:10:57.447976 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:10:57.448192 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:10:57.448251 3798194 disk_worker.go:723] disk space is not compliant W1122 01:10:57.448255 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:10:57.451017 3798194 disk_worker.go:509] system disk:vda1 I1122 01:10:57.451054 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 01:10:58.448198 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 01:10:58.448222 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 01:10:59.448538 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 01:10:59.449615 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:10:59.449665 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:10:59.449685 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:10:59.471187 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:11:01.002850 3798194 disk_info.go:125] begin check local disk info of client I1122 01:11:01.006405 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:11:01.006413 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0700 0xc0002e0740] E1122 01:11:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:11:06.411980 3798194 memory.go:184] no items to output this cycle I1122 01:11:06.412139 3798194 cpu.go:275] no items to output this cycle E1122 01:11:16.411972 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:11:16.411992 3798194 memory.go:184] no items to output this cycle I1122 01:11:16.412168 3798194 cpu.go:275] no items to output this cycle E1122 01:11:26.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:11:26.412898 3798194 memory.go:191] Add success. I1122 01:11:26.416938 3798194 cpu.go:282] Add success. I1122 01:11:26.441471 3798194 net.go:642] Add success. I1122 01:11:26.443122 3798194 net.go:778] primary dev: ETH0 I1122 01:11:26.443137 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:11:26.443149 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:11:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:11:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:11:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:11:36.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:11:36.411982 3798194 memory.go:184] no items to output this cycle I1122 01:11:36.412135 3798194 cpu.go:275] no items to output this cycle E1122 01:11:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:11:46.411973 3798194 memory.go:184] no items to output this cycle I1122 01:11:46.412134 3798194 cpu.go:275] no items to output this cycle E1122 01:11:56.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:11:56.412261 3798194 memory.go:191] Add success. W1122 01:11:56.412294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:11:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:11:56.412309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:11:56.416663 3798194 cpu.go:282] Add success. I1122 01:11:56.440956 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:11:56.441552 3798194 net.go:642] Add success. I1122 01:11:56.442942 3798194 net.go:778] primary dev: ETH0 I1122 01:11:56.442957 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:11:56.442971 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:11:57.452143 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:11:57.452312 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:11:57.452368 3798194 disk_worker.go:723] disk space is not compliant W1122 01:11:57.452371 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:11:57.473784 3798194 disk_worker.go:509] system disk:vda1 I1122 01:11:57.473822 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:11:58.446048 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:11:59.450033 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:11:59.450132 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:11:59.450158 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:11:59.471645 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:12:01.008149 3798194 disk_info.go:125] begin check local disk info of client I1122 01:12:01.014765 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:12:01.014775 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040f840 0xc00040f880] E1122 01:12:06.411971 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:12:06.411994 3798194 memory.go:184] no items to output this cycle I1122 01:12:06.412157 3798194 cpu.go:275] no items to output this cycle E1122 01:12:16.413467 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:12:16.413490 3798194 memory.go:184] no items to output this cycle I1122 01:12:16.413642 3798194 cpu.go:275] no items to output this cycle I1122 01:12:17.585435 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:12:17.585444 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:12:26.411505 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:12:26.411543 3798194 memory.go:191] Add success. I1122 01:12:26.411895 3798194 cpu.go:282] Add success. I1122 01:12:26.441653 3798194 net.go:642] Add success. I1122 01:12:26.446049 3798194 net.go:778] primary dev: ETH0 I1122 01:12:26.446064 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:12:26.446078 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:12:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:12:29.450130 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:12:29.450160 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:12:36.412587 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:12:36.412612 3798194 memory.go:184] no items to output this cycle I1122 01:12:36.412756 3798194 cpu.go:275] no items to output this cycle E1122 01:12:46.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:12:46.412266 3798194 memory.go:184] no items to output this cycle I1122 01:12:46.412424 3798194 cpu.go:275] no items to output this cycle E1122 01:12:56.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:12:56.412274 3798194 memory.go:191] Add success. W1122 01:12:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:12:56.412323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:12:56.412326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:12:56.416747 3798194 cpu.go:282] Add success. I1122 01:12:56.441425 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:12:56.441887 3798194 net.go:642] Add success. I1122 01:12:56.443408 3798194 net.go:778] primary dev: ETH0 I1122 01:12:56.443423 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:12:56.443435 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:12:57.450347 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:12:57.450587 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:12:57.450599 3798194 disk_worker.go:723] disk space is not compliant W1122 01:12:57.450602 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:12:57.460252 3798194 disk_worker.go:509] system disk:vda1 I1122 01:12:57.460296 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:12:58.446011 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:12:59.450032 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:12:59.450121 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:12:59.450150 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:12:59.471660 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:13:01.014877 3798194 disk_info.go:125] begin check local disk info of client I1122 01:13:01.019085 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:13:01.019094 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003da300 0xc0003da340] E1122 01:13:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:13:06.411978 3798194 memory.go:184] no items to output this cycle I1122 01:13:06.412136 3798194 cpu.go:275] no items to output this cycle E1122 01:13:16.412250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:13:16.412277 3798194 memory.go:184] no items to output this cycle I1122 01:13:16.412425 3798194 cpu.go:275] no items to output this cycle E1122 01:13:26.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:13:26.411951 3798194 memory.go:191] Add success. I1122 01:13:26.412494 3798194 cpu.go:282] Add success. I1122 01:13:26.441218 3798194 net.go:642] Add success. I1122 01:13:26.445522 3798194 net.go:778] primary dev: ETH0 I1122 01:13:26.445537 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:13:26.445550 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:13:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:13:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:13:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:13:36.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:13:36.412874 3798194 memory.go:184] no items to output this cycle I1122 01:13:36.413030 3798194 cpu.go:275] no items to output this cycle E1122 01:13:46.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:13:46.412873 3798194 memory.go:184] no items to output this cycle I1122 01:13:46.413019 3798194 cpu.go:275] no items to output this cycle E1122 01:13:56.412843 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:13:56.412875 3798194 memory.go:191] Add success. W1122 01:13:56.412906 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:13:56.412917 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:13:56.412920 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:13:56.417605 3798194 cpu.go:282] Add success. I1122 01:13:56.441241 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:13:56.441710 3798194 net.go:642] Add success. I1122 01:13:56.443152 3798194 net.go:778] primary dev: ETH0 I1122 01:13:56.443168 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:13:56.443180 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:13:57.450685 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:13:57.450911 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:13:57.450924 3798194 disk_worker.go:723] disk space is not compliant W1122 01:13:57.450927 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:13:57.459029 3798194 disk_worker.go:509] system disk:vda1 I1122 01:13:57.459072 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:13:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:13:59.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:13:59.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:13:59.450495 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:13:59.471011 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:14:01.021146 3798194 disk_info.go:125] begin check local disk info of client I1122 01:14:01.026847 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:14:01.026857 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000268840 0xc000268880] E1122 01:14:06.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:14:06.412264 3798194 memory.go:184] no items to output this cycle I1122 01:14:06.412419 3798194 cpu.go:275] no items to output this cycle E1122 01:14:16.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:14:16.412889 3798194 memory.go:184] no items to output this cycle I1122 01:14:16.413040 3798194 cpu.go:275] no items to output this cycle E1122 01:14:26.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:14:26.411958 3798194 memory.go:191] Add success. I1122 01:14:26.412323 3798194 cpu.go:282] Add success. I1122 01:14:26.441796 3798194 net.go:642] Add success. I1122 01:14:26.445720 3798194 net.go:778] primary dev: ETH0 I1122 01:14:26.445735 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:14:26.445748 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:14:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:14:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:14:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:14:36.413166 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:14:36.413190 3798194 memory.go:184] no items to output this cycle I1122 01:14:36.413343 3798194 cpu.go:275] no items to output this cycle E1122 01:14:46.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:14:46.412249 3798194 memory.go:184] no items to output this cycle I1122 01:14:46.412393 3798194 cpu.go:275] no items to output this cycle E1122 01:14:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:14:56.411957 3798194 memory.go:191] Add success. W1122 01:14:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:14:56.412005 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:14:56.412008 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:14:56.412387 3798194 cpu.go:282] Add success. I1122 01:14:56.441847 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:14:56.442456 3798194 net.go:642] Add success. I1122 01:14:56.446315 3798194 net.go:778] primary dev: ETH0 I1122 01:14:56.446331 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:14:56.446343 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:14:57.451562 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:14:57.451731 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:14:57.451786 3798194 disk_worker.go:723] disk space is not compliant W1122 01:14:57.451789 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:14:57.464882 3798194 disk_worker.go:509] system disk:vda1 I1122 01:14:57.464932 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:14:58.445987 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:14:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:14:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:14:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:14:59.471659 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:15:01.029149 3798194 disk_info.go:125] begin check local disk info of client I1122 01:15:01.032745 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:15:01.032754 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0180 0xc0002e01c0] E1122 01:15:06.412408 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:15:06.412431 3798194 memory.go:184] no items to output this cycle I1122 01:15:06.412574 3798194 cpu.go:275] no items to output this cycle E1122 01:15:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:15:16.411978 3798194 memory.go:184] no items to output this cycle I1122 01:15:16.412147 3798194 cpu.go:275] no items to output this cycle I1122 01:15:17.585617 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:15:17.585626 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:15:26.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:15:26.413261 3798194 memory.go:191] Add success. I1122 01:15:26.417744 3798194 cpu.go:282] Add success. I1122 01:15:26.442254 3798194 net.go:642] Add success. I1122 01:15:26.443797 3798194 net.go:778] primary dev: ETH0 I1122 01:15:26.443812 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:15:26.443825 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:15:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:15:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:15:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:15:36.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:15:36.413265 3798194 memory.go:184] no items to output this cycle I1122 01:15:36.413406 3798194 cpu.go:275] no items to output this cycle E1122 01:15:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:15:46.411976 3798194 memory.go:184] no items to output this cycle I1122 01:15:46.412132 3798194 cpu.go:275] no items to output this cycle E1122 01:15:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:15:56.411961 3798194 memory.go:191] Add success. W1122 01:15:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:15:56.412008 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:15:56.412012 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:15:56.412334 3798194 cpu.go:282] Add success. I1122 01:15:56.441695 3798194 event_worker.go:152] Polling the log file for events... I1122 01:15:56.445961 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:15:56.446463 3798194 net.go:642] Add success. I1122 01:15:56.450811 3798194 net.go:778] primary dev: ETH0 I1122 01:15:56.450827 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:15:56.450846 3798194 net.go:659] Add eni metrics success, count: 7 E1122 01:15:57.449250 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 01:15:57.449264 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:15:57.449494 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:15:57.449503 3798194 disk_worker.go:723] disk space is not compliant W1122 01:15:57.449507 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:15:57.451329 3798194 disk_worker.go:509] system disk:vda1 I1122 01:15:57.451369 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 01:15:58.448147 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 01:15:58.448163 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 01:15:59.448984 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 01:15:59.450057 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:15:59.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:15:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:15:59.471647 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:16:01.032861 3798194 disk_info.go:125] begin check local disk info of client I1122 01:16:01.036716 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:16:01.036724 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004567c0 0xc000456800] E1122 01:16:06.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:16:06.412875 3798194 memory.go:184] no items to output this cycle I1122 01:16:06.413016 3798194 cpu.go:275] no items to output this cycle E1122 01:16:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:16:16.412262 3798194 memory.go:184] no items to output this cycle I1122 01:16:16.412415 3798194 cpu.go:275] no items to output this cycle E1122 01:16:26.412032 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:16:26.412067 3798194 memory.go:191] Add success. I1122 01:16:26.416426 3798194 cpu.go:282] Add success. I1122 01:16:26.442096 3798194 net.go:642] Add success. I1122 01:16:26.443679 3798194 net.go:778] primary dev: ETH0 I1122 01:16:26.443694 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:16:26.443708 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:16:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:16:29.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:16:29.450149 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:16:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:16:36.411978 3798194 memory.go:184] no items to output this cycle I1122 01:16:36.412131 3798194 cpu.go:275] no items to output this cycle E1122 01:16:46.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:16:46.411987 3798194 memory.go:184] no items to output this cycle I1122 01:16:46.412157 3798194 cpu.go:275] no items to output this cycle E1122 01:16:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:16:56.411949 3798194 memory.go:191] Add success. W1122 01:16:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:16:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:16:56.412000 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:16:56.412354 3798194 cpu.go:282] Add success. I1122 01:16:56.441940 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:16:56.442544 3798194 net.go:642] Add success. I1122 01:16:56.446737 3798194 net.go:778] primary dev: ETH0 I1122 01:16:56.446752 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:16:56.446763 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:16:57.450297 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:16:57.450504 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:16:57.450515 3798194 disk_worker.go:723] disk space is not compliant W1122 01:16:57.450519 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:16:57.458577 3798194 disk_worker.go:509] system disk:vda1 I1122 01:16:57.458618 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:16:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:16:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:16:59.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:16:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:16:59.471663 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:17:01.036821 3798194 disk_info.go:125] begin check local disk info of client I1122 01:17:01.040401 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:17:01.040410 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000381a00 0xc000381a40] E1122 01:17:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:17:06.412259 3798194 memory.go:184] no items to output this cycle I1122 01:17:06.412401 3798194 cpu.go:275] no items to output this cycle E1122 01:17:16.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:17:16.412256 3798194 memory.go:184] no items to output this cycle I1122 01:17:16.412410 3798194 cpu.go:275] no items to output this cycle E1122 01:17:26.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:17:26.412268 3798194 memory.go:191] Add success. I1122 01:17:26.416522 3798194 cpu.go:282] Add success. I1122 01:17:26.441284 3798194 net.go:642] Add success. I1122 01:17:26.442792 3798194 net.go:778] primary dev: ETH0 I1122 01:17:26.442805 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:17:26.442819 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:17:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:17:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:17:29.450147 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:17:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:17:36.412257 3798194 memory.go:184] no items to output this cycle I1122 01:17:36.412400 3798194 cpu.go:275] no items to output this cycle E1122 01:17:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:17:46.411975 3798194 memory.go:184] no items to output this cycle I1122 01:17:46.412134 3798194 cpu.go:275] no items to output this cycle E1122 01:17:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:17:56.411949 3798194 memory.go:191] Add success. W1122 01:17:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:17:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:17:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:17:56.412324 3798194 cpu.go:282] Add success. I1122 01:17:56.441817 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:17:56.442316 3798194 net.go:642] Add success. I1122 01:17:56.446459 3798194 net.go:778] primary dev: ETH0 I1122 01:17:56.446476 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:17:56.446489 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:17:57.453628 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:17:57.453803 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:17:57.453860 3798194 disk_worker.go:723] disk space is not compliant W1122 01:17:57.453864 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:17:57.462592 3798194 disk_worker.go:509] system disk:vda1 I1122 01:17:57.462644 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:17:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:17:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:17:59.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:17:59.450510 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:17:59.471016 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:18:01.040515 3798194 disk_info.go:125] begin check local disk info of client I1122 01:18:01.044492 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:18:01.044501 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002f9dc0 0xc0002f9e00] E1122 01:18:06.412182 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:18:06.412206 3798194 memory.go:184] no items to output this cycle I1122 01:18:06.412352 3798194 cpu.go:275] no items to output this cycle E1122 01:18:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:18:16.411981 3798194 memory.go:184] no items to output this cycle I1122 01:18:16.412141 3798194 cpu.go:275] no items to output this cycle I1122 01:18:17.585725 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:18:17.585734 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:18:26.411563 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:18:26.411595 3798194 memory.go:191] Add success. I1122 01:18:26.411934 3798194 cpu.go:282] Add success. I1122 01:18:26.441553 3798194 net.go:642] Add success. I1122 01:18:26.445685 3798194 net.go:778] primary dev: ETH0 I1122 01:18:26.445700 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:18:26.445714 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:18:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:18:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:18:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:18:36.412601 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:18:36.412623 3798194 memory.go:184] no items to output this cycle I1122 01:18:36.412765 3798194 cpu.go:275] no items to output this cycle E1122 01:18:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:18:46.411983 3798194 memory.go:184] no items to output this cycle I1122 01:18:46.412139 3798194 cpu.go:275] no items to output this cycle E1122 01:18:56.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:18:56.412369 3798194 memory.go:191] Add success. W1122 01:18:56.412409 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:18:56.412421 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:18:56.412425 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:18:56.417666 3798194 cpu.go:282] Add success. I1122 01:18:56.441570 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:18:56.442033 3798194 net.go:642] Add success. I1122 01:18:56.443687 3798194 net.go:778] primary dev: ETH0 I1122 01:18:56.443702 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:18:56.443715 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:18:57.445318 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:18:57.445611 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:18:57.445623 3798194 disk_worker.go:723] disk space is not compliant W1122 01:18:57.445627 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:18:57.448079 3798194 disk_worker.go:509] system disk:vda1 I1122 01:18:57.448133 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:18:58.445982 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:18:59.450496 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:18:59.450587 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:18:59.450614 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:18:59.471132 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:19:01.044603 3798194 disk_info.go:125] begin check local disk info of client I1122 01:19:01.048579 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:19:01.048589 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030f700 0xc00030f740] E1122 01:19:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:19:06.411975 3798194 memory.go:184] no items to output this cycle I1122 01:19:06.412134 3798194 cpu.go:275] no items to output this cycle E1122 01:19:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:19:16.411976 3798194 memory.go:184] no items to output this cycle I1122 01:19:16.412144 3798194 cpu.go:275] no items to output this cycle E1122 01:19:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:19:26.411953 3798194 memory.go:191] Add success. I1122 01:19:26.412308 3798194 cpu.go:282] Add success. I1122 01:19:26.442062 3798194 net.go:642] Add success. I1122 01:19:26.446014 3798194 net.go:778] primary dev: ETH0 I1122 01:19:26.446028 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:19:26.446040 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:19:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:19:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:19:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:19:36.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:19:36.411987 3798194 memory.go:184] no items to output this cycle I1122 01:19:36.412140 3798194 cpu.go:275] no items to output this cycle E1122 01:19:46.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:19:46.412887 3798194 memory.go:184] no items to output this cycle I1122 01:19:46.413034 3798194 cpu.go:275] no items to output this cycle E1122 01:19:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:19:56.411949 3798194 memory.go:191] Add success. W1122 01:19:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:19:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:19:56.412000 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:19:56.412322 3798194 cpu.go:282] Add success. I1122 01:19:56.441942 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:19:56.442540 3798194 net.go:642] Add success. I1122 01:19:56.446923 3798194 net.go:778] primary dev: ETH0 I1122 01:19:56.446939 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:19:56.446955 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:19:57.450386 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:19:57.450587 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:19:57.450597 3798194 disk_worker.go:723] disk space is not compliant W1122 01:19:57.450601 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:19:57.458635 3798194 disk_worker.go:509] system disk:vda1 I1122 01:19:57.458691 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:19:58.446012 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:19:59.450409 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:19:59.450493 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:19:59.450520 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:19:59.471022 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:20:01.048692 3798194 disk_info.go:125] begin check local disk info of client I1122 01:20:01.052685 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:20:01.052694 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fe600 0xc0003fe640] E1122 01:20:06.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:20:06.413269 3798194 memory.go:184] no items to output this cycle I1122 01:20:06.413414 3798194 cpu.go:275] no items to output this cycle E1122 01:20:16.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:20:16.411990 3798194 memory.go:184] no items to output this cycle I1122 01:20:16.412150 3798194 cpu.go:275] no items to output this cycle E1122 01:20:26.412843 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:20:26.412876 3798194 memory.go:191] Add success. I1122 01:20:26.417470 3798194 cpu.go:282] Add success. I1122 01:20:26.442244 3798194 net.go:642] Add success. I1122 01:20:26.443856 3798194 net.go:778] primary dev: ETH0 I1122 01:20:26.443870 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:20:26.443883 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:20:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:20:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:20:29.450125 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:20:36.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:20:36.411985 3798194 memory.go:184] no items to output this cycle I1122 01:20:36.412137 3798194 cpu.go:275] no items to output this cycle E1122 01:20:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:20:46.411970 3798194 memory.go:184] no items to output this cycle I1122 01:20:46.412128 3798194 cpu.go:275] no items to output this cycle E1122 01:20:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:20:56.411955 3798194 memory.go:191] Add success. W1122 01:20:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:20:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:20:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:20:56.412358 3798194 cpu.go:282] Add success. I1122 01:20:56.444499 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:20:56.444976 3798194 net.go:642] Add success. I1122 01:20:56.449293 3798194 net.go:778] primary dev: ETH0 I1122 01:20:56.449310 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:20:56.449324 3798194 net.go:659] Add eni metrics success, count: 7 E1122 01:20:57.450337 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 01:20:57.450350 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:20:57.450552 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:20:57.450561 3798194 disk_worker.go:723] disk space is not compliant W1122 01:20:57.450564 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:20:57.459241 3798194 disk_worker.go:509] system disk:vda1 I1122 01:20:57.459282 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 01:20:58.447775 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 01:20:58.447789 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 01:20:59.449507 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 01:20:59.450580 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:20:59.450631 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:20:59.450650 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:20:59.471185 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:21:01.052787 3798194 disk_info.go:125] begin check local disk info of client I1122 01:21:01.056475 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:21:01.056484 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d8300 0xc0003d8340] E1122 01:21:06.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:21:06.413253 3798194 memory.go:184] no items to output this cycle I1122 01:21:06.413404 3798194 cpu.go:275] no items to output this cycle E1122 01:21:16.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:21:16.412267 3798194 memory.go:184] no items to output this cycle I1122 01:21:16.412420 3798194 cpu.go:275] no items to output this cycle I1122 01:21:17.586209 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:21:17.586218 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:21:26.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:21:26.412886 3798194 memory.go:191] Add success. I1122 01:21:26.417261 3798194 cpu.go:282] Add success. I1122 01:21:26.441294 3798194 net.go:642] Add success. I1122 01:21:26.442868 3798194 net.go:778] primary dev: ETH0 I1122 01:21:26.442882 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:21:26.442895 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:21:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:21:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:21:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:21:36.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:21:36.411970 3798194 memory.go:184] no items to output this cycle I1122 01:21:36.412127 3798194 cpu.go:275] no items to output this cycle E1122 01:21:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:21:46.411975 3798194 memory.go:184] no items to output this cycle I1122 01:21:46.412133 3798194 cpu.go:275] no items to output this cycle E1122 01:21:56.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:21:56.413268 3798194 memory.go:191] Add success. W1122 01:21:56.413301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:21:56.413313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:21:56.413316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:21:56.417974 3798194 cpu.go:282] Add success. I1122 01:21:56.441353 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:21:56.441837 3798194 net.go:642] Add success. I1122 01:21:56.443398 3798194 net.go:778] primary dev: ETH0 I1122 01:21:56.443413 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:21:56.443424 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:21:57.449782 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:21:57.450030 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:21:57.450041 3798194 disk_worker.go:723] disk space is not compliant W1122 01:21:57.450044 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:21:57.451858 3798194 disk_worker.go:509] system disk:vda1 I1122 01:21:57.451896 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:21:58.446415 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:21:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:21:59.450489 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:21:59.450515 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:21:59.471020 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:22:01.056593 3798194 disk_info.go:125] begin check local disk info of client I1122 01:22:01.060446 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:22:01.060455 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00026d6c0 0xc00026d700] E1122 01:22:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:22:06.411982 3798194 memory.go:184] no items to output this cycle I1122 01:22:06.412137 3798194 cpu.go:275] no items to output this cycle E1122 01:22:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:22:16.411981 3798194 memory.go:184] no items to output this cycle I1122 01:22:16.412139 3798194 cpu.go:275] no items to output this cycle E1122 01:22:26.412872 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:22:26.412905 3798194 memory.go:191] Add success. I1122 01:22:26.417243 3798194 cpu.go:282] Add success. I1122 01:22:26.441749 3798194 net.go:642] Add success. I1122 01:22:26.443274 3798194 net.go:778] primary dev: ETH0 I1122 01:22:26.443289 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:22:26.443303 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:22:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:22:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:22:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:22:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:22:36.411974 3798194 memory.go:184] no items to output this cycle I1122 01:22:36.412123 3798194 cpu.go:275] no items to output this cycle E1122 01:22:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:22:46.411983 3798194 memory.go:184] no items to output this cycle I1122 01:22:46.412145 3798194 cpu.go:275] no items to output this cycle E1122 01:22:56.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:22:56.412892 3798194 memory.go:191] Add success. W1122 01:22:56.412924 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:22:56.412935 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:22:56.412939 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:22:56.417404 3798194 cpu.go:282] Add success. I1122 01:22:56.441429 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:22:56.441894 3798194 net.go:642] Add success. I1122 01:22:56.443538 3798194 net.go:778] primary dev: ETH0 I1122 01:22:56.443553 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:22:56.443564 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:22:57.446281 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:22:57.446560 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:22:57.446571 3798194 disk_worker.go:723] disk space is not compliant W1122 01:22:57.446575 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:22:57.451569 3798194 disk_worker.go:509] system disk:vda1 I1122 01:22:57.451612 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:22:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:22:59.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:22:59.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:22:59.450497 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:22:59.471033 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:23:01.060556 3798194 disk_info.go:125] begin check local disk info of client I1122 01:23:01.064436 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:23:01.064445 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ee80 0xc00030eec0] E1122 01:23:06.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:23:06.413266 3798194 memory.go:184] no items to output this cycle I1122 01:23:06.413409 3798194 cpu.go:275] no items to output this cycle E1122 01:23:16.411354 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:23:16.411383 3798194 memory.go:184] no items to output this cycle I1122 01:23:16.411538 3798194 cpu.go:275] no items to output this cycle E1122 01:23:26.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:23:26.413278 3798194 memory.go:191] Add success. I1122 01:23:26.417581 3798194 cpu.go:282] Add success. I1122 01:23:26.442127 3798194 net.go:642] Add success. I1122 01:23:26.443589 3798194 net.go:778] primary dev: ETH0 I1122 01:23:26.443605 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:23:26.443617 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:23:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:23:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:23:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:23:36.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:23:36.412258 3798194 memory.go:184] no items to output this cycle I1122 01:23:36.412400 3798194 cpu.go:275] no items to output this cycle E1122 01:23:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:23:46.411978 3798194 memory.go:184] no items to output this cycle I1122 01:23:46.412138 3798194 cpu.go:275] no items to output this cycle E1122 01:23:56.413008 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:23:56.413042 3798194 memory.go:191] Add success. W1122 01:23:56.413077 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:23:56.413088 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:23:56.413092 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:23:56.417707 3798194 cpu.go:282] Add success. I1122 01:23:56.441127 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:23:56.441630 3798194 net.go:642] Add success. I1122 01:23:56.443226 3798194 net.go:778] primary dev: ETH0 I1122 01:23:56.443241 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:23:56.443254 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:23:57.450738 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:23:57.450955 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:23:57.450966 3798194 disk_worker.go:723] disk space is not compliant W1122 01:23:57.450969 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:23:57.460506 3798194 disk_worker.go:509] system disk:vda1 I1122 01:23:57.460549 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:23:58.446384 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:23:59.450595 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:23:59.450681 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:23:59.450707 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:23:59.471232 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:24:01.067136 3798194 disk_info.go:125] begin check local disk info of client I1122 01:24:01.071542 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:24:01.071553 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041a000 0xc00041a040] E1122 01:24:06.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:24:06.412263 3798194 memory.go:184] no items to output this cycle I1122 01:24:06.412404 3798194 cpu.go:275] no items to output this cycle E1122 01:24:16.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:24:16.411998 3798194 memory.go:184] no items to output this cycle I1122 01:24:16.412164 3798194 cpu.go:275] no items to output this cycle I1122 01:24:17.586384 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:24:17.586393 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:24:26.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:24:26.412275 3798194 memory.go:191] Add success. I1122 01:24:26.417159 3798194 cpu.go:282] Add success. I1122 01:24:26.441728 3798194 net.go:642] Add success. I1122 01:24:26.443380 3798194 net.go:778] primary dev: ETH0 I1122 01:24:26.443395 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:24:26.443408 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:24:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:24:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:24:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:24:36.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:24:36.412262 3798194 memory.go:184] no items to output this cycle I1122 01:24:36.412407 3798194 cpu.go:275] no items to output this cycle E1122 01:24:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:24:46.411978 3798194 memory.go:184] no items to output this cycle I1122 01:24:46.412138 3798194 cpu.go:275] no items to output this cycle E1122 01:24:56.411933 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:24:56.411966 3798194 memory.go:191] Add success. W1122 01:24:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:24:56.412012 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:24:56.412017 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:24:56.412364 3798194 cpu.go:282] Add success. I1122 01:24:56.441805 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:24:56.442319 3798194 net.go:642] Add success. I1122 01:24:56.446263 3798194 net.go:778] primary dev: ETH0 I1122 01:24:56.446281 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:24:56.446293 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:24:57.450687 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:24:57.450910 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:24:57.450923 3798194 disk_worker.go:723] disk space is not compliant W1122 01:24:57.450926 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:24:57.464013 3798194 disk_worker.go:509] system disk:vda1 I1122 01:24:57.464065 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:24:58.445985 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:24:59.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:24:59.450128 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:24:59.450155 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:24:59.471650 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:25:01.071659 3798194 disk_info.go:125] begin check local disk info of client I1122 01:25:01.075317 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:25:01.075327 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037d400 0xc00037d440] E1122 01:25:06.412960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:25:06.412982 3798194 memory.go:184] no items to output this cycle I1122 01:25:06.413154 3798194 cpu.go:275] no items to output this cycle E1122 01:25:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:25:16.411976 3798194 memory.go:184] no items to output this cycle I1122 01:25:16.412138 3798194 cpu.go:275] no items to output this cycle E1122 01:25:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:25:26.413263 3798194 memory.go:191] Add success. I1122 01:25:26.417479 3798194 cpu.go:282] Add success. I1122 01:25:26.442251 3798194 net.go:642] Add success. I1122 01:25:26.443803 3798194 net.go:778] primary dev: ETH0 I1122 01:25:26.443818 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:25:26.443832 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:25:29.450580 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:25:29.450669 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:25:29.450696 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:25:36.412007 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:25:36.412030 3798194 memory.go:184] no items to output this cycle I1122 01:25:36.412204 3798194 cpu.go:275] no items to output this cycle E1122 01:25:46.411975 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:25:46.411998 3798194 memory.go:184] no items to output this cycle I1122 01:25:46.412159 3798194 cpu.go:275] no items to output this cycle E1122 01:25:56.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:25:56.412265 3798194 memory.go:191] Add success. W1122 01:25:56.412298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:25:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:25:56.412313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:25:56.416891 3798194 cpu.go:282] Add success. I1122 01:25:56.443771 3798194 event_worker.go:152] Polling the log file for events... I1122 01:25:56.443987 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:25:56.444573 3798194 net.go:642] Add success. I1122 01:25:56.446214 3798194 net.go:778] primary dev: ETH0 I1122 01:25:56.446228 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:25:56.446242 3798194 net.go:659] Add eni metrics success, count: 7 E1122 01:25:57.450651 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 01:25:57.450667 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:25:57.450870 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:25:57.450879 3798194 disk_worker.go:723] disk space is not compliant W1122 01:25:57.450882 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:25:57.461849 3798194 disk_worker.go:509] system disk:vda1 I1122 01:25:57.461892 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 01:25:58.448150 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 01:25:58.448167 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 01:25:59.448989 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 01:25:59.450066 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:25:59.450126 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:25:59.450156 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:25:59.471679 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:26:01.078144 3798194 disk_info.go:125] begin check local disk info of client I1122 01:26:01.081961 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:26:01.081970 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000445e40 0xc000445e80] E1122 01:26:06.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:26:06.413267 3798194 memory.go:184] no items to output this cycle I1122 01:26:06.413417 3798194 cpu.go:275] no items to output this cycle E1122 01:26:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:26:16.411981 3798194 memory.go:184] no items to output this cycle I1122 01:26:16.412150 3798194 cpu.go:275] no items to output this cycle E1122 01:26:26.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:26:26.412884 3798194 memory.go:191] Add success. I1122 01:26:26.417146 3798194 cpu.go:282] Add success. I1122 01:26:26.441762 3798194 net.go:642] Add success. I1122 01:26:26.443308 3798194 net.go:778] primary dev: ETH0 I1122 01:26:26.443323 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:26:26.443335 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:26:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:26:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:26:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:26:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:26:36.413257 3798194 memory.go:184] no items to output this cycle I1122 01:26:36.413399 3798194 cpu.go:275] no items to output this cycle E1122 01:26:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:26:46.413254 3798194 memory.go:184] no items to output this cycle I1122 01:26:46.413405 3798194 cpu.go:275] no items to output this cycle E1122 01:26:56.411943 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:26:56.411975 3798194 memory.go:191] Add success. W1122 01:26:56.412006 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:26:56.412019 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:26:56.412024 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:26:56.412370 3798194 cpu.go:282] Add success. I1122 01:26:56.441829 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:26:56.442464 3798194 net.go:642] Add success. I1122 01:26:56.446663 3798194 net.go:778] primary dev: ETH0 I1122 01:26:56.446680 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:26:56.446696 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:26:57.451001 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:26:57.451221 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:26:57.451283 3798194 disk_worker.go:723] disk space is not compliant W1122 01:26:57.451287 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:26:57.459732 3798194 disk_worker.go:509] system disk:vda1 I1122 01:26:57.459774 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:26:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:26:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:26:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:26:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:26:59.471648 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:27:01.082079 3798194 disk_info.go:125] begin check local disk info of client I1122 01:27:01.086000 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:27:01.086009 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000478a80 0xc000478ac0] E1122 01:27:06.412771 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:27:06.412792 3798194 memory.go:184] no items to output this cycle I1122 01:27:06.412946 3798194 cpu.go:275] no items to output this cycle E1122 01:27:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:27:16.411978 3798194 memory.go:184] no items to output this cycle I1122 01:27:16.412132 3798194 cpu.go:275] no items to output this cycle I1122 01:27:17.586557 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:27:17.586566 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:27:26.411236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:27:26.411271 3798194 memory.go:191] Add success. I1122 01:27:26.415593 3798194 cpu.go:282] Add success. I1122 01:27:26.441438 3798194 net.go:642] Add success. I1122 01:27:26.443091 3798194 net.go:778] primary dev: ETH0 I1122 01:27:26.443106 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:27:26.443131 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:27:29.450029 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:27:29.450121 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:27:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:27:36.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:27:36.412889 3798194 memory.go:184] no items to output this cycle I1122 01:27:36.413034 3798194 cpu.go:275] no items to output this cycle E1122 01:27:46.412250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:27:46.412272 3798194 memory.go:184] no items to output this cycle I1122 01:27:46.412419 3798194 cpu.go:275] no items to output this cycle E1122 01:27:56.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:27:56.412275 3798194 memory.go:191] Add success. W1122 01:27:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:27:56.412319 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:27:56.412322 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:27:56.416655 3798194 cpu.go:282] Add success. I1122 01:27:56.441082 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:27:56.441661 3798194 net.go:642] Add success. I1122 01:27:56.443271 3798194 net.go:778] primary dev: ETH0 I1122 01:27:56.443286 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:27:56.443298 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:27:57.450693 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:27:57.450898 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:27:57.450909 3798194 disk_worker.go:723] disk space is not compliant W1122 01:27:57.450912 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:27:57.459041 3798194 disk_worker.go:509] system disk:vda1 I1122 01:27:57.459085 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:27:58.446031 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:27:59.450131 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:27:59.450207 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:27:59.450232 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:27:59.470690 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:28:01.086103 3798194 disk_info.go:125] begin check local disk info of client I1122 01:28:01.089933 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:28:01.089946 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00044a000 0xc00044a040] E1122 01:28:06.411371 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:28:06.411393 3798194 memory.go:184] no items to output this cycle I1122 01:28:06.411536 3798194 cpu.go:275] no items to output this cycle E1122 01:28:16.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:28:16.412887 3798194 memory.go:184] no items to output this cycle I1122 01:28:16.413033 3798194 cpu.go:275] no items to output this cycle E1122 01:28:26.411929 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:28:26.411968 3798194 memory.go:191] Add success. I1122 01:28:26.412343 3798194 cpu.go:282] Add success. I1122 01:28:26.441970 3798194 net.go:642] Add success. I1122 01:28:26.446002 3798194 net.go:778] primary dev: ETH0 I1122 01:28:26.446018 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:28:26.446032 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:28:29.450048 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:28:29.450142 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:28:29.450173 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:28:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:28:36.412256 3798194 memory.go:184] no items to output this cycle I1122 01:28:36.412403 3798194 cpu.go:275] no items to output this cycle E1122 01:28:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:28:46.411976 3798194 memory.go:184] no items to output this cycle I1122 01:28:46.412138 3798194 cpu.go:275] no items to output this cycle E1122 01:28:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:28:56.412268 3798194 memory.go:191] Add success. W1122 01:28:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:28:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:28:56.412317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:28:56.416586 3798194 cpu.go:282] Add success. I1122 01:28:56.440957 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:28:56.441561 3798194 net.go:642] Add success. I1122 01:28:56.443165 3798194 net.go:778] primary dev: ETH0 I1122 01:28:56.443181 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:28:56.443192 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:28:57.445918 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:28:57.446147 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:28:57.446223 3798194 disk_worker.go:723] disk space is not compliant W1122 01:28:57.446227 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:28:57.451282 3798194 disk_worker.go:509] system disk:vda1 I1122 01:28:57.451323 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:28:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:28:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:28:59.450137 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:28:59.450163 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:28:59.471652 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:29:01.090053 3798194 disk_info.go:125] begin check local disk info of client I1122 01:29:01.093634 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:29:01.093644 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030d480 0xc00030d4c0] E1122 01:29:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:29:06.411981 3798194 memory.go:184] no items to output this cycle I1122 01:29:06.412141 3798194 cpu.go:275] no items to output this cycle E1122 01:29:16.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:29:16.412254 3798194 memory.go:184] no items to output this cycle I1122 01:29:16.412402 3798194 cpu.go:275] no items to output this cycle E1122 01:29:26.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:29:26.412891 3798194 memory.go:191] Add success. I1122 01:29:26.417177 3798194 cpu.go:282] Add success. I1122 01:29:26.441672 3798194 net.go:642] Add success. I1122 01:29:26.443242 3798194 net.go:778] primary dev: ETH0 I1122 01:29:26.443255 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:29:26.443268 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:29:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:29:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:29:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:29:36.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:29:36.413258 3798194 memory.go:184] no items to output this cycle I1122 01:29:36.413412 3798194 cpu.go:275] no items to output this cycle E1122 01:29:46.413285 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:29:46.413316 3798194 memory.go:184] no items to output this cycle I1122 01:29:46.413517 3798194 cpu.go:275] no items to output this cycle E1122 01:29:56.414978 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:29:56.415023 3798194 memory.go:191] Add success. W1122 01:29:56.415061 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:29:56.415074 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:29:56.415077 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:29:56.419996 3798194 cpu.go:282] Add success. I1122 01:29:56.440960 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:29:56.441565 3798194 net.go:642] Add success. I1122 01:29:56.443251 3798194 net.go:778] primary dev: ETH0 I1122 01:29:56.443266 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:29:56.443279 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:29:57.451059 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:29:57.451293 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:29:57.451305 3798194 disk_worker.go:723] disk space is not compliant W1122 01:29:57.451308 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:29:57.460160 3798194 disk_worker.go:509] system disk:vda1 I1122 01:29:57.460202 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:29:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:29:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:29:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:29:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:29:59.471647 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:30:01.096157 3798194 disk_info.go:125] begin check local disk info of client I1122 01:30:01.099375 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:30:01.099384 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ed00 0xc00030ed40] E1122 01:30:06.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:30:06.412267 3798194 memory.go:184] no items to output this cycle I1122 01:30:06.412408 3798194 cpu.go:275] no items to output this cycle E1122 01:30:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:30:16.411974 3798194 memory.go:184] no items to output this cycle I1122 01:30:16.412138 3798194 cpu.go:275] no items to output this cycle I1122 01:30:17.586732 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:30:17.586741 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:30:26.411319 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:30:26.411355 3798194 memory.go:191] Add success. I1122 01:30:26.411704 3798194 cpu.go:282] Add success. I1122 01:30:26.441275 3798194 net.go:642] Add success. I1122 01:30:26.445324 3798194 net.go:778] primary dev: ETH0 I1122 01:30:26.445340 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:30:26.445352 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:30:29.450445 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:30:29.450520 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:30:29.450545 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:30:36.412337 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:30:36.412358 3798194 memory.go:184] no items to output this cycle I1122 01:30:36.412500 3798194 cpu.go:275] no items to output this cycle E1122 01:30:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:30:46.412261 3798194 memory.go:184] no items to output this cycle I1122 01:30:46.412408 3798194 cpu.go:275] no items to output this cycle E1122 01:30:56.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:30:56.412891 3798194 memory.go:191] Add success. W1122 01:30:56.412919 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:30:56.412930 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:30:56.412933 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:30:56.417750 3798194 cpu.go:282] Add success. I1122 01:30:56.444577 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:30:56.445078 3798194 net.go:642] Add success. I1122 01:30:56.447698 3798194 net.go:778] primary dev: ETH0 I1122 01:30:56.447715 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:30:56.447728 3798194 net.go:659] Add eni metrics success, count: 7 E1122 01:30:57.445234 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 01:30:57.445251 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:30:57.445452 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:30:57.445513 3798194 disk_worker.go:723] disk space is not compliant W1122 01:30:57.445516 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:30:57.447978 3798194 disk_worker.go:509] system disk:vda1 I1122 01:30:57.448016 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 01:30:58.447821 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 01:30:58.447836 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 01:30:59.448650 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 01:30:59.449726 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:30:59.449775 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:30:59.449801 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:30:59.471334 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:31:01.099483 3798194 disk_info.go:125] begin check local disk info of client I1122 01:31:01.103511 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:31:01.103521 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000492240 0xc000492280] E1122 01:31:06.412524 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:31:06.412622 3798194 memory.go:184] no items to output this cycle I1122 01:31:06.413098 3798194 cpu.go:275] no items to output this cycle E1122 01:31:16.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:31:16.412872 3798194 memory.go:184] no items to output this cycle I1122 01:31:16.413025 3798194 cpu.go:275] no items to output this cycle E1122 01:31:26.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:31:26.412902 3798194 memory.go:191] Add success. I1122 01:31:26.417439 3798194 cpu.go:282] Add success. I1122 01:31:26.442066 3798194 net.go:642] Add success. I1122 01:31:26.443605 3798194 net.go:778] primary dev: ETH0 I1122 01:31:26.443631 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:31:26.443644 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:31:29.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:31:29.450465 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:31:29.450495 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:31:36.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:31:36.412269 3798194 memory.go:184] no items to output this cycle I1122 01:31:36.412410 3798194 cpu.go:275] no items to output this cycle E1122 01:31:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:31:46.412256 3798194 memory.go:184] no items to output this cycle I1122 01:31:46.412406 3798194 cpu.go:275] no items to output this cycle E1122 01:31:56.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:31:56.412284 3798194 memory.go:191] Add success. W1122 01:31:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:31:56.412325 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:31:56.412328 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:31:56.416724 3798194 cpu.go:282] Add success. I1122 01:31:56.441132 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:31:56.441609 3798194 net.go:642] Add success. I1122 01:31:56.443165 3798194 net.go:778] primary dev: ETH0 I1122 01:31:56.443181 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:31:56.443193 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:31:57.450950 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:31:57.451200 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:31:57.451210 3798194 disk_worker.go:723] disk space is not compliant W1122 01:31:57.451214 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:31:57.460018 3798194 disk_worker.go:509] system disk:vda1 I1122 01:31:57.460058 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:31:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:31:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:31:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:31:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:31:59.471640 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:32:01.106243 3798194 disk_info.go:125] begin check local disk info of client I1122 01:32:01.109922 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:32:01.109932 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037d340 0xc00037d380] E1122 01:32:06.411985 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:32:06.412007 3798194 memory.go:184] no items to output this cycle I1122 01:32:06.412165 3798194 cpu.go:275] no items to output this cycle E1122 01:32:16.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:32:16.412267 3798194 memory.go:184] no items to output this cycle I1122 01:32:16.412411 3798194 cpu.go:275] no items to output this cycle E1122 01:32:26.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:32:26.411958 3798194 memory.go:191] Add success. I1122 01:32:26.412321 3798194 cpu.go:282] Add success. I1122 01:32:26.441938 3798194 net.go:642] Add success. I1122 01:32:26.446263 3798194 net.go:778] primary dev: ETH0 I1122 01:32:26.446288 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:32:26.446308 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:32:29.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:32:29.450118 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:32:29.450148 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:32:36.412771 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:32:36.412796 3798194 memory.go:184] no items to output this cycle I1122 01:32:36.412949 3798194 cpu.go:275] no items to output this cycle E1122 01:32:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:32:46.411982 3798194 memory.go:184] no items to output this cycle I1122 01:32:46.412146 3798194 cpu.go:275] no items to output this cycle E1122 01:32:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:32:56.411949 3798194 memory.go:191] Add success. W1122 01:32:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:32:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:32:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:32:56.412333 3798194 cpu.go:282] Add success. I1122 01:32:56.441776 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:32:56.442289 3798194 net.go:642] Add success. I1122 01:32:56.446202 3798194 net.go:778] primary dev: ETH0 I1122 01:32:56.446217 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:32:56.446229 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:32:57.451317 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:32:57.451546 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:32:57.451557 3798194 disk_worker.go:723] disk space is not compliant W1122 01:32:57.451560 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:32:57.460652 3798194 disk_worker.go:509] system disk:vda1 I1122 01:32:57.460696 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:32:58.446540 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:32:59.450645 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:32:59.450729 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:32:59.450753 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:32:59.471192 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:33:01.113283 3798194 disk_info.go:125] begin check local disk info of client I1122 01:33:01.116801 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:33:01.116810 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c5740 0xc0003c5780] E1122 01:33:06.413173 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:33:06.413198 3798194 memory.go:184] no items to output this cycle I1122 01:33:06.413349 3798194 cpu.go:275] no items to output this cycle E1122 01:33:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:33:16.413264 3798194 memory.go:184] no items to output this cycle I1122 01:33:16.413425 3798194 cpu.go:275] no items to output this cycle I1122 01:33:17.586898 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:33:17.586907 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:33:26.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:33:26.412276 3798194 memory.go:191] Add success. I1122 01:33:26.416291 3798194 cpu.go:282] Add success. I1122 01:33:26.441749 3798194 net.go:642] Add success. I1122 01:33:26.443257 3798194 net.go:778] primary dev: ETH0 I1122 01:33:26.443270 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:33:26.443283 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:33:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:33:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:33:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:33:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:33:36.411978 3798194 memory.go:184] no items to output this cycle I1122 01:33:36.412138 3798194 cpu.go:275] no items to output this cycle E1122 01:33:46.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:33:46.412256 3798194 memory.go:184] no items to output this cycle I1122 01:33:46.412407 3798194 cpu.go:275] no items to output this cycle E1122 01:33:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:33:56.411957 3798194 memory.go:191] Add success. W1122 01:33:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:33:56.412003 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:33:56.412007 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:33:56.412347 3798194 cpu.go:282] Add success. I1122 01:33:56.442039 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:33:56.442988 3798194 net.go:642] Add success. I1122 01:33:56.447181 3798194 net.go:778] primary dev: ETH0 I1122 01:33:56.447197 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:33:56.447214 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:33:57.450591 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:33:57.450804 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:33:57.450815 3798194 disk_worker.go:723] disk space is not compliant W1122 01:33:57.450818 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:33:57.458982 3798194 disk_worker.go:509] system disk:vda1 I1122 01:33:57.459020 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:33:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:33:59.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:33:59.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:33:59.450508 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:33:59.471058 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:34:01.116923 3798194 disk_info.go:125] begin check local disk info of client I1122 01:34:01.120472 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:34:01.120481 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00048f980 0xc00048f9c0] E1122 01:34:06.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:34:06.413259 3798194 memory.go:184] no items to output this cycle I1122 01:34:06.413414 3798194 cpu.go:275] no items to output this cycle E1122 01:34:16.411979 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:34:16.412002 3798194 memory.go:184] no items to output this cycle I1122 01:34:16.412164 3798194 cpu.go:275] no items to output this cycle E1122 01:34:26.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:34:26.411944 3798194 memory.go:191] Add success. I1122 01:34:26.412323 3798194 cpu.go:282] Add success. I1122 01:34:26.441923 3798194 net.go:642] Add success. I1122 01:34:26.445756 3798194 net.go:778] primary dev: ETH0 I1122 01:34:26.445773 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:34:26.445786 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:34:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:34:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:34:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:34:36.413155 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:34:36.413177 3798194 memory.go:184] no items to output this cycle I1122 01:34:36.413322 3798194 cpu.go:275] no items to output this cycle E1122 01:34:46.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:34:46.413259 3798194 memory.go:184] no items to output this cycle I1122 01:34:46.413406 3798194 cpu.go:275] no items to output this cycle E1122 01:34:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:34:56.411948 3798194 memory.go:191] Add success. W1122 01:34:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:34:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:34:56.411996 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:34:56.412338 3798194 cpu.go:282] Add success. I1122 01:34:56.441796 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:34:56.442420 3798194 net.go:642] Add success. I1122 01:34:56.446628 3798194 net.go:778] primary dev: ETH0 I1122 01:34:56.446643 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:34:56.446656 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:34:57.453180 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:34:57.453405 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:34:57.453416 3798194 disk_worker.go:723] disk space is not compliant W1122 01:34:57.453419 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:34:57.467840 3798194 disk_worker.go:509] system disk:vda1 I1122 01:34:57.467884 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:34:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:34:59.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:34:59.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:34:59.450501 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:34:59.471016 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:35:01.120574 3798194 disk_info.go:125] begin check local disk info of client I1122 01:35:01.124426 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:35:01.124434 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00042e9c0 0xc00042ea00] E1122 01:35:06.413357 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:35:06.413381 3798194 memory.go:184] no items to output this cycle I1122 01:35:06.413646 3798194 cpu.go:275] no items to output this cycle E1122 01:35:16.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:35:16.411987 3798194 memory.go:184] no items to output this cycle I1122 01:35:16.412153 3798194 cpu.go:275] no items to output this cycle E1122 01:35:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:35:26.411950 3798194 memory.go:191] Add success. I1122 01:35:26.412325 3798194 cpu.go:282] Add success. I1122 01:35:26.441926 3798194 net.go:642] Add success. I1122 01:35:26.445995 3798194 net.go:778] primary dev: ETH0 I1122 01:35:26.446011 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:35:26.446026 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:35:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:35:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:35:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:35:36.413211 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:35:36.413236 3798194 memory.go:184] no items to output this cycle I1122 01:35:36.413378 3798194 cpu.go:275] no items to output this cycle E1122 01:35:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:35:46.411979 3798194 memory.go:184] no items to output this cycle I1122 01:35:46.412141 3798194 cpu.go:275] no items to output this cycle E1122 01:35:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:35:56.411950 3798194 memory.go:191] Add success. W1122 01:35:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:35:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:35:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:35:56.412333 3798194 cpu.go:282] Add success. I1122 01:35:56.441560 3798194 event_worker.go:152] Polling the log file for events... I1122 01:35:56.463316 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:35:56.463825 3798194 net.go:642] Add success. I1122 01:35:56.473453 3798194 net.go:778] primary dev: ETH0 I1122 01:35:56.473470 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:35:56.473483 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:35:56.473510 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795280756, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 01:35:56.481365 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 01:35:57.450359 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 01:35:57.450373 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:35:57.450573 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:35:57.450582 3798194 disk_worker.go:723] disk space is not compliant W1122 01:35:57.450585 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:35:57.463799 3798194 disk_worker.go:509] system disk:vda1 I1122 01:35:57.463841 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 01:35:58.447762 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 01:35:58.447776 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 01:35:59.448941 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 01:35:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:35:59.450065 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:35:59.450087 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:35:59.471612 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:36:01.127145 3798194 disk_info.go:125] begin check local disk info of client I1122 01:36:01.130998 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:36:01.131007 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00038d780 0xc00038d7c0] E1122 01:36:06.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:36:06.413268 3798194 memory.go:184] no items to output this cycle I1122 01:36:06.413416 3798194 cpu.go:275] no items to output this cycle E1122 01:36:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:36:16.411985 3798194 memory.go:184] no items to output this cycle I1122 01:36:16.412144 3798194 cpu.go:275] no items to output this cycle I1122 01:36:17.587073 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:36:17.587082 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:36:26.411278 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:36:26.411310 3798194 memory.go:191] Add success. I1122 01:36:26.411648 3798194 cpu.go:282] Add success. I1122 01:36:26.441666 3798194 net.go:642] Add success. I1122 01:36:26.446262 3798194 net.go:778] primary dev: ETH0 I1122 01:36:26.446277 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:36:26.446290 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:36:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:36:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:36:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:36:36.412312 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:36:36.412335 3798194 memory.go:184] no items to output this cycle I1122 01:36:36.412488 3798194 cpu.go:275] no items to output this cycle E1122 01:36:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:36:46.413259 3798194 memory.go:184] no items to output this cycle I1122 01:36:46.413408 3798194 cpu.go:275] no items to output this cycle E1122 01:36:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:36:56.411949 3798194 memory.go:191] Add success. W1122 01:36:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:36:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:36:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:36:56.412353 3798194 cpu.go:282] Add success. I1122 01:36:56.441820 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:36:56.442335 3798194 net.go:642] Add success. I1122 01:36:56.446616 3798194 net.go:778] primary dev: ETH0 I1122 01:36:56.446631 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:36:56.446643 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:36:57.450454 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:36:57.450674 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:36:57.450685 3798194 disk_worker.go:723] disk space is not compliant W1122 01:36:57.450688 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:36:57.459227 3798194 disk_worker.go:509] system disk:vda1 I1122 01:36:57.459274 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:36:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:36:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:36:59.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:36:59.450504 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:36:59.471040 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:37:01.131118 3798194 disk_info.go:125] begin check local disk info of client I1122 01:37:01.135281 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:37:01.135291 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00038d980 0xc00038d9c0] E1122 01:37:06.412106 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:37:06.412146 3798194 memory.go:184] no items to output this cycle I1122 01:37:06.412312 3798194 cpu.go:275] no items to output this cycle E1122 01:37:16.412348 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:37:16.412381 3798194 memory.go:184] no items to output this cycle I1122 01:37:16.412543 3798194 cpu.go:275] no items to output this cycle E1122 01:37:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:37:26.411951 3798194 memory.go:191] Add success. I1122 01:37:26.412332 3798194 cpu.go:282] Add success. I1122 01:37:26.441957 3798194 net.go:642] Add success. I1122 01:37:26.445951 3798194 net.go:778] primary dev: ETH0 I1122 01:37:26.445966 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:37:26.445980 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:37:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:37:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:37:29.450144 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:37:36.412922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:37:36.412944 3798194 memory.go:184] no items to output this cycle I1122 01:37:36.413083 3798194 cpu.go:275] no items to output this cycle E1122 01:37:46.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:37:46.411967 3798194 memory.go:184] no items to output this cycle I1122 01:37:46.412121 3798194 cpu.go:275] no items to output this cycle E1122 01:37:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:37:56.412060 3798194 memory.go:191] Add success. W1122 01:37:56.412098 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:37:56.412124 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:37:56.412130 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:37:56.412701 3798194 cpu.go:282] Add success. I1122 01:37:56.440959 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:37:56.441544 3798194 net.go:642] Add success. I1122 01:37:56.445547 3798194 net.go:778] primary dev: ETH0 I1122 01:37:56.445561 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:37:56.445574 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:37:57.450738 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:37:57.450914 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:37:57.450968 3798194 disk_worker.go:723] disk space is not compliant W1122 01:37:57.450971 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:37:57.460169 3798194 disk_worker.go:509] system disk:vda1 I1122 01:37:57.460211 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:37:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:37:59.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:37:59.450493 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:37:59.450520 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:37:59.471039 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:38:01.135400 3798194 disk_info.go:125] begin check local disk info of client I1122 01:38:01.138940 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:38:01.138949 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ef40 0xc00030ef80] E1122 01:38:06.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:38:06.411974 3798194 memory.go:184] no items to output this cycle I1122 01:38:06.412138 3798194 cpu.go:275] no items to output this cycle E1122 01:38:16.411991 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:38:16.412016 3798194 memory.go:184] no items to output this cycle I1122 01:38:16.412183 3798194 cpu.go:275] no items to output this cycle E1122 01:38:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:38:26.412270 3798194 memory.go:191] Add success. I1122 01:38:26.416633 3798194 cpu.go:282] Add success. I1122 01:38:26.442179 3798194 net.go:642] Add success. I1122 01:38:26.443614 3798194 net.go:778] primary dev: ETH0 I1122 01:38:26.443629 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:38:26.443642 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:38:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:38:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:38:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:38:36.412563 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:38:36.412585 3798194 memory.go:184] no items to output this cycle I1122 01:38:36.412728 3798194 cpu.go:275] no items to output this cycle E1122 01:38:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:38:46.411977 3798194 memory.go:184] no items to output this cycle I1122 01:38:46.412139 3798194 cpu.go:275] no items to output this cycle E1122 01:38:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:38:56.413263 3798194 memory.go:191] Add success. W1122 01:38:56.413296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:38:56.413309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:38:56.413312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:38:56.418223 3798194 cpu.go:282] Add success. I1122 01:38:56.441580 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:38:56.442157 3798194 net.go:642] Add success. I1122 01:38:56.443740 3798194 net.go:778] primary dev: ETH0 I1122 01:38:56.443756 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:38:56.443769 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:38:57.447985 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:38:57.448213 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:38:57.448273 3798194 disk_worker.go:723] disk space is not compliant W1122 01:38:57.448276 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:38:57.451058 3798194 disk_worker.go:509] system disk:vda1 I1122 01:38:57.451098 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:38:58.446389 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:38:59.450386 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:38:59.450465 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:38:59.450495 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:38:59.471014 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:39:01.139054 3798194 disk_info.go:125] begin check local disk info of client I1122 01:39:01.142614 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:39:01.142623 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004161c0 0xc000416200] E1122 01:39:06.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:39:06.413253 3798194 memory.go:184] no items to output this cycle I1122 01:39:06.413395 3798194 cpu.go:275] no items to output this cycle E1122 01:39:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:39:16.413258 3798194 memory.go:184] no items to output this cycle I1122 01:39:16.413409 3798194 cpu.go:275] no items to output this cycle I1122 01:39:17.587232 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:39:17.587242 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:39:26.411218 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:39:26.411249 3798194 memory.go:191] Add success. I1122 01:39:26.411582 3798194 cpu.go:282] Add success. I1122 01:39:26.442193 3798194 net.go:642] Add success. I1122 01:39:26.446082 3798194 net.go:778] primary dev: ETH0 I1122 01:39:26.446096 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:39:26.446123 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:39:29.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:39:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:39:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:39:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:39:36.412251 3798194 memory.go:184] no items to output this cycle I1122 01:39:36.412391 3798194 cpu.go:275] no items to output this cycle E1122 01:39:46.411938 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:39:46.411959 3798194 memory.go:184] no items to output this cycle I1122 01:39:46.412125 3798194 cpu.go:275] no items to output this cycle E1122 01:39:56.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:39:56.413265 3798194 memory.go:191] Add success. W1122 01:39:56.413292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:39:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:39:56.413307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:39:56.417337 3798194 cpu.go:282] Add success. I1122 01:39:56.441743 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:39:56.442327 3798194 net.go:642] Add success. I1122 01:39:56.443842 3798194 net.go:778] primary dev: ETH0 I1122 01:39:56.443856 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:39:56.443867 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:39:57.450373 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:39:57.450548 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:39:57.450604 3798194 disk_worker.go:723] disk space is not compliant W1122 01:39:57.450607 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:39:57.458729 3798194 disk_worker.go:509] system disk:vda1 I1122 01:39:57.458772 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:39:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:39:59.450105 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:39:59.450187 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:39:59.450213 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:39:59.470687 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:40:01.142720 3798194 disk_info.go:125] begin check local disk info of client I1122 01:40:01.146642 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:40:01.146652 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003857c0 0xc000385800] E1122 01:40:06.411975 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:40:06.411998 3798194 memory.go:184] no items to output this cycle I1122 01:40:06.412152 3798194 cpu.go:275] no items to output this cycle E1122 01:40:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:40:16.411976 3798194 memory.go:184] no items to output this cycle I1122 01:40:16.412133 3798194 cpu.go:275] no items to output this cycle E1122 01:40:26.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:40:26.412261 3798194 memory.go:191] Add success. I1122 01:40:26.416703 3798194 cpu.go:282] Add success. I1122 01:40:26.442244 3798194 net.go:642] Add success. I1122 01:40:26.443861 3798194 net.go:778] primary dev: ETH0 I1122 01:40:26.443877 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:40:26.443906 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:40:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:40:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:40:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:40:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:40:36.412252 3798194 memory.go:184] no items to output this cycle I1122 01:40:36.412391 3798194 cpu.go:275] no items to output this cycle E1122 01:40:46.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:40:46.412881 3798194 memory.go:184] no items to output this cycle I1122 01:40:46.413023 3798194 cpu.go:275] no items to output this cycle E1122 01:40:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:40:56.411960 3798194 memory.go:191] Add success. W1122 01:40:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:40:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:40:56.412002 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:40:56.412333 3798194 cpu.go:282] Add success. I1122 01:40:56.444411 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:40:56.444891 3798194 net.go:642] Add success. I1122 01:40:56.449093 3798194 net.go:778] primary dev: ETH0 I1122 01:40:56.449108 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:40:56.449133 3798194 net.go:659] Add eni metrics success, count: 7 E1122 01:40:57.450302 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 01:40:57.450316 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:40:57.450508 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:40:57.450518 3798194 disk_worker.go:723] disk space is not compliant W1122 01:40:57.450522 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:40:57.458587 3798194 disk_worker.go:509] system disk:vda1 I1122 01:40:57.458634 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 01:40:58.447768 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 01:40:58.447784 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 01:40:59.448601 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 01:40:59.449676 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:40:59.449726 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:40:59.449744 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:40:59.471251 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:41:01.146757 3798194 disk_info.go:125] begin check local disk info of client I1122 01:41:01.150302 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:41:01.150310 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00025cec0 0xc00025cf00] E1122 01:41:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:41:06.412258 3798194 memory.go:184] no items to output this cycle I1122 01:41:06.412412 3798194 cpu.go:275] no items to output this cycle E1122 01:41:16.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:41:16.412881 3798194 memory.go:184] no items to output this cycle I1122 01:41:16.413031 3798194 cpu.go:275] no items to output this cycle E1122 01:41:26.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:41:26.411953 3798194 memory.go:191] Add success. I1122 01:41:26.412311 3798194 cpu.go:282] Add success. I1122 01:41:26.441897 3798194 net.go:642] Add success. I1122 01:41:26.445905 3798194 net.go:778] primary dev: ETH0 I1122 01:41:26.445921 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:41:26.445934 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:41:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:41:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:41:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:41:36.412754 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:41:36.412775 3798194 memory.go:184] no items to output this cycle I1122 01:41:36.412925 3798194 cpu.go:275] no items to output this cycle E1122 01:41:46.411943 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:41:46.411977 3798194 memory.go:184] no items to output this cycle I1122 01:41:46.412144 3798194 cpu.go:275] no items to output this cycle E1122 01:41:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:41:56.411950 3798194 memory.go:191] Add success. W1122 01:41:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:41:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:41:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:41:56.412338 3798194 cpu.go:282] Add success. I1122 01:41:56.441800 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:41:56.442745 3798194 net.go:642] Add success. I1122 01:41:56.447091 3798194 net.go:778] primary dev: ETH0 I1122 01:41:56.447107 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:41:56.447134 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:41:57.450588 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:41:57.450785 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:41:57.450795 3798194 disk_worker.go:723] disk space is not compliant W1122 01:41:57.450798 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:41:57.459169 3798194 disk_worker.go:509] system disk:vda1 I1122 01:41:57.459208 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:41:58.446644 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:41:59.449668 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:41:59.449747 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:41:59.449773 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:41:59.471281 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:42:01.150412 3798194 disk_info.go:125] begin check local disk info of client I1122 01:42:01.154264 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:42:01.154273 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a9ac0 0xc0004a9b00] E1122 01:42:06.412761 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:42:06.412791 3798194 memory.go:184] no items to output this cycle I1122 01:42:06.412942 3798194 cpu.go:275] no items to output this cycle E1122 01:42:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:42:16.411971 3798194 memory.go:184] no items to output this cycle I1122 01:42:16.412123 3798194 cpu.go:275] no items to output this cycle I1122 01:42:17.587411 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:42:17.587419 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:42:26.411211 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:42:26.411242 3798194 memory.go:191] Add success. I1122 01:42:26.411568 3798194 cpu.go:282] Add success. I1122 01:42:26.442180 3798194 net.go:642] Add success. I1122 01:42:26.446085 3798194 net.go:778] primary dev: ETH0 I1122 01:42:26.446104 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:42:26.446133 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:42:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:42:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:42:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:42:36.412224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:42:36.412245 3798194 memory.go:184] no items to output this cycle I1122 01:42:36.412398 3798194 cpu.go:275] no items to output this cycle E1122 01:42:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:42:46.411972 3798194 memory.go:184] no items to output this cycle I1122 01:42:46.412121 3798194 cpu.go:275] no items to output this cycle E1122 01:42:56.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:42:56.411943 3798194 memory.go:191] Add success. W1122 01:42:56.411969 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:42:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:42:56.411985 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:42:56.412333 3798194 cpu.go:282] Add success. I1122 01:42:56.441796 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:42:56.442327 3798194 net.go:642] Add success. I1122 01:42:56.446271 3798194 net.go:778] primary dev: ETH0 I1122 01:42:56.446286 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:42:56.446298 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:42:57.450296 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:42:57.450452 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:42:57.450507 3798194 disk_worker.go:723] disk space is not compliant W1122 01:42:57.450510 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:42:57.458597 3798194 disk_worker.go:509] system disk:vda1 I1122 01:42:57.458640 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:42:58.446464 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:42:59.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:42:59.450466 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:42:59.450491 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:42:59.471002 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:43:01.154368 3798194 disk_info.go:125] begin check local disk info of client I1122 01:43:01.157926 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:43:01.157934 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004048c0 0xc000404900] E1122 01:43:06.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:43:06.413253 3798194 memory.go:184] no items to output this cycle I1122 01:43:06.413392 3798194 cpu.go:275] no items to output this cycle E1122 01:43:16.411975 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:43:16.411997 3798194 memory.go:184] no items to output this cycle I1122 01:43:16.412252 3798194 cpu.go:275] no items to output this cycle E1122 01:43:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:43:26.411955 3798194 memory.go:191] Add success. I1122 01:43:26.412334 3798194 cpu.go:282] Add success. I1122 01:43:26.441878 3798194 net.go:642] Add success. I1122 01:43:26.445858 3798194 net.go:778] primary dev: ETH0 I1122 01:43:26.445873 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:43:26.445887 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:43:29.450031 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:43:29.450129 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:43:29.450159 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:43:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:43:36.412253 3798194 memory.go:184] no items to output this cycle I1122 01:43:36.412396 3798194 cpu.go:275] no items to output this cycle E1122 01:43:46.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:43:46.412879 3798194 memory.go:184] no items to output this cycle I1122 01:43:46.413026 3798194 cpu.go:275] no items to output this cycle E1122 01:43:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:43:56.411945 3798194 memory.go:191] Add success. W1122 01:43:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:43:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:43:56.411993 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:43:56.412330 3798194 cpu.go:282] Add success. I1122 01:43:56.441804 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:43:56.442304 3798194 net.go:642] Add success. I1122 01:43:56.446313 3798194 net.go:778] primary dev: ETH0 I1122 01:43:56.446327 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:43:56.446340 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:43:57.451038 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:43:57.451224 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:43:57.451285 3798194 disk_worker.go:723] disk space is not compliant W1122 01:43:57.451288 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:43:57.463077 3798194 disk_worker.go:509] system disk:vda1 I1122 01:43:57.463151 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:43:58.446390 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:43:59.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:43:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:43:59.450147 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:43:59.471662 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:44:01.158064 3798194 disk_info.go:125] begin check local disk info of client I1122 01:44:01.161662 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:44:01.161672 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000345440 0xc000345480] E1122 01:44:06.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:44:06.412253 3798194 memory.go:184] no items to output this cycle I1122 01:44:06.412393 3798194 cpu.go:275] no items to output this cycle E1122 01:44:16.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:44:16.411986 3798194 memory.go:184] no items to output this cycle I1122 01:44:16.412145 3798194 cpu.go:275] no items to output this cycle E1122 01:44:26.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:44:26.412893 3798194 memory.go:191] Add success. I1122 01:44:26.417234 3798194 cpu.go:282] Add success. I1122 01:44:26.441757 3798194 net.go:642] Add success. I1122 01:44:26.443454 3798194 net.go:778] primary dev: ETH0 I1122 01:44:26.443468 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:44:26.443481 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:44:29.450202 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:44:29.450284 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:44:29.450309 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:44:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:44:36.411972 3798194 memory.go:184] no items to output this cycle I1122 01:44:36.412140 3798194 cpu.go:275] no items to output this cycle E1122 01:44:46.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:44:46.412260 3798194 memory.go:184] no items to output this cycle I1122 01:44:46.412412 3798194 cpu.go:275] no items to output this cycle E1122 01:44:56.411932 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:44:56.411970 3798194 memory.go:191] Add success. W1122 01:44:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:44:56.412016 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:44:56.412019 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:44:56.412376 3798194 cpu.go:282] Add success. I1122 01:44:56.443166 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:44:56.443788 3798194 net.go:642] Add success. I1122 01:44:56.447458 3798194 net.go:778] primary dev: ETH0 I1122 01:44:56.447473 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:44:56.447485 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:44:57.445313 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:44:57.445605 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:44:57.445619 3798194 disk_worker.go:723] disk space is not compliant W1122 01:44:57.445622 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:44:57.448099 3798194 disk_worker.go:509] system disk:vda1 I1122 01:44:57.448162 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:44:58.445987 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:44:59.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:44:59.450124 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:44:59.450152 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:44:59.471651 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:45:01.161801 3798194 disk_info.go:125] begin check local disk info of client I1122 01:45:01.165433 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:45:01.165442 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e8e00 0xc0002e8e40] E1122 01:45:06.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:45:06.412265 3798194 memory.go:184] no items to output this cycle I1122 01:45:06.412417 3798194 cpu.go:275] no items to output this cycle E1122 01:45:16.411997 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:45:16.412020 3798194 memory.go:184] no items to output this cycle I1122 01:45:16.412179 3798194 cpu.go:275] no items to output this cycle I1122 01:45:17.587589 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:45:17.587596 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:45:26.411328 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:45:26.411361 3798194 memory.go:191] Add success. I1122 01:45:26.411694 3798194 cpu.go:282] Add success. I1122 01:45:26.441311 3798194 net.go:642] Add success. I1122 01:45:26.445529 3798194 net.go:778] primary dev: ETH0 I1122 01:45:26.445543 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:45:26.445556 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:45:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:45:29.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:45:29.450129 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:45:36.412815 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:45:36.412838 3798194 memory.go:184] no items to output this cycle I1122 01:45:36.412980 3798194 cpu.go:275] no items to output this cycle E1122 01:45:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:45:46.411975 3798194 memory.go:184] no items to output this cycle I1122 01:45:46.412147 3798194 cpu.go:275] no items to output this cycle E1122 01:45:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:45:56.411942 3798194 memory.go:191] Add success. W1122 01:45:56.411970 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:45:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:45:56.411986 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:45:56.412316 3798194 cpu.go:282] Add success. I1122 01:45:56.441553 3798194 event_worker.go:152] Polling the log file for events... I1122 01:45:56.444347 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:45:56.444955 3798194 net.go:642] Add success. I1122 01:45:56.449009 3798194 net.go:778] primary dev: ETH0 I1122 01:45:56.449025 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:45:56.449039 3798194 net.go:659] Add eni metrics success, count: 7 E1122 01:45:57.450590 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 01:45:57.450603 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:45:57.450790 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:45:57.450800 3798194 disk_worker.go:723] disk space is not compliant W1122 01:45:57.450803 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:45:57.459128 3798194 disk_worker.go:509] system disk:vda1 I1122 01:45:57.459172 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 01:45:58.448723 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 01:45:58.448737 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 01:45:59.448984 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 01:45:59.450060 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:45:59.450125 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:45:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:45:59.471654 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:46:01.169168 3798194 disk_info.go:125] begin check local disk info of client I1122 01:46:01.172671 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:46:01.172680 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004762c0 0xc000476300] E1122 01:46:06.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:46:06.413248 3798194 memory.go:184] no items to output this cycle I1122 01:46:06.413388 3798194 cpu.go:275] no items to output this cycle E1122 01:46:16.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:46:16.411987 3798194 memory.go:184] no items to output this cycle I1122 01:46:16.412144 3798194 cpu.go:275] no items to output this cycle E1122 01:46:26.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:46:26.411950 3798194 memory.go:191] Add success. I1122 01:46:26.412297 3798194 cpu.go:282] Add success. I1122 01:46:26.441745 3798194 net.go:642] Add success. I1122 01:46:26.445779 3798194 net.go:778] primary dev: ETH0 I1122 01:46:26.445795 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:46:26.445808 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:46:29.450249 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:46:29.450327 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:46:29.450352 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:46:36.413162 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:46:36.413184 3798194 memory.go:184] no items to output this cycle I1122 01:46:36.413329 3798194 cpu.go:275] no items to output this cycle E1122 01:46:46.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:46:46.411970 3798194 memory.go:184] no items to output this cycle I1122 01:46:46.412131 3798194 cpu.go:275] no items to output this cycle E1122 01:46:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:46:56.411955 3798194 memory.go:191] Add success. W1122 01:46:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:46:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:46:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:46:56.412337 3798194 cpu.go:282] Add success. I1122 01:46:56.441783 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:46:56.442403 3798194 net.go:642] Add success. I1122 01:46:56.446282 3798194 net.go:778] primary dev: ETH0 I1122 01:46:56.446303 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:46:56.446322 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:46:57.450820 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:46:57.450988 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:46:57.451040 3798194 disk_worker.go:723] disk space is not compliant W1122 01:46:57.451044 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:46:57.459871 3798194 disk_worker.go:509] system disk:vda1 I1122 01:46:57.459911 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:46:58.446493 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:46:59.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:46:59.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:46:59.450508 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:46:59.471036 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:47:01.172775 3798194 disk_info.go:125] begin check local disk info of client I1122 01:47:01.176358 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:47:01.176368 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000387880 0xc0003878c0] E1122 01:47:06.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:47:06.413253 3798194 memory.go:184] no items to output this cycle I1122 01:47:06.413404 3798194 cpu.go:275] no items to output this cycle E1122 01:47:16.411995 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:47:16.412017 3798194 memory.go:184] no items to output this cycle I1122 01:47:16.412178 3798194 cpu.go:275] no items to output this cycle E1122 01:47:26.411910 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:47:26.411945 3798194 memory.go:191] Add success. I1122 01:47:26.412309 3798194 cpu.go:282] Add success. I1122 01:47:26.441753 3798194 net.go:642] Add success. I1122 01:47:26.446000 3798194 net.go:778] primary dev: ETH0 I1122 01:47:26.446015 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:47:26.446029 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:47:29.450415 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:47:29.450498 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:47:29.450529 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:47:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:47:36.412258 3798194 memory.go:184] no items to output this cycle I1122 01:47:36.412617 3798194 cpu.go:275] no items to output this cycle E1122 01:47:46.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:47:46.412264 3798194 memory.go:184] no items to output this cycle I1122 01:47:46.412410 3798194 cpu.go:275] no items to output this cycle E1122 01:47:56.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:47:56.413278 3798194 memory.go:191] Add success. W1122 01:47:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:47:56.413324 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:47:56.413335 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:47:56.417892 3798194 cpu.go:282] Add success. I1122 01:47:56.441311 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:47:56.441789 3798194 net.go:642] Add success. I1122 01:47:56.443346 3798194 net.go:778] primary dev: ETH0 I1122 01:47:56.443363 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:47:56.443375 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:47:57.446313 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:47:57.446596 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:47:57.446610 3798194 disk_worker.go:723] disk space is not compliant W1122 01:47:57.446613 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:47:57.451666 3798194 disk_worker.go:509] system disk:vda1 I1122 01:47:57.451714 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:47:58.446377 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:47:59.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:47:59.450497 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:47:59.450523 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:47:59.471063 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:48:01.179616 3798194 disk_info.go:125] begin check local disk info of client I1122 01:48:01.183205 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:48:01.183214 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003eadc0 0xc0003eae00] E1122 01:48:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:48:06.411980 3798194 memory.go:184] no items to output this cycle I1122 01:48:06.412136 3798194 cpu.go:275] no items to output this cycle E1122 01:48:16.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:48:16.412871 3798194 memory.go:184] no items to output this cycle I1122 01:48:16.413019 3798194 cpu.go:275] no items to output this cycle I1122 01:48:17.587716 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:48:17.587725 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:48:26.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:48:26.413279 3798194 memory.go:191] Add success. I1122 01:48:26.417883 3798194 cpu.go:282] Add success. I1122 01:48:26.441410 3798194 net.go:642] Add success. I1122 01:48:26.442784 3798194 net.go:778] primary dev: ETH0 I1122 01:48:26.442799 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:48:26.442812 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:48:29.450408 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:48:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:48:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:48:36.413422 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:48:36.413446 3798194 memory.go:184] no items to output this cycle I1122 01:48:36.413591 3798194 cpu.go:275] no items to output this cycle E1122 01:48:46.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:48:46.413260 3798194 memory.go:184] no items to output this cycle I1122 01:48:46.413402 3798194 cpu.go:275] no items to output this cycle E1122 01:48:56.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:48:56.413283 3798194 memory.go:191] Add success. W1122 01:48:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:48:56.413324 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:48:56.413327 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:48:56.417388 3798194 cpu.go:282] Add success. I1122 01:48:56.441772 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:48:56.442274 3798194 net.go:642] Add success. I1122 01:48:56.443637 3798194 net.go:778] primary dev: ETH0 I1122 01:48:56.443652 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:48:56.443665 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:48:57.451089 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:48:57.451325 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:48:57.451336 3798194 disk_worker.go:723] disk space is not compliant W1122 01:48:57.451339 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:48:57.461247 3798194 disk_worker.go:509] system disk:vda1 I1122 01:48:57.461289 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:48:58.445688 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:48:59.449673 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:48:59.449744 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:48:59.449769 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:48:59.471306 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:49:01.186146 3798194 disk_info.go:125] begin check local disk info of client I1122 01:49:01.189787 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:49:01.189797 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a5900 0xc0004a5940] E1122 01:49:06.412841 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:49:06.412863 3798194 memory.go:184] no items to output this cycle I1122 01:49:06.413010 3798194 cpu.go:275] no items to output this cycle E1122 01:49:16.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:49:16.413251 3798194 memory.go:184] no items to output this cycle I1122 01:49:16.413395 3798194 cpu.go:275] no items to output this cycle E1122 01:49:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:49:26.413267 3798194 memory.go:191] Add success. I1122 01:49:26.417738 3798194 cpu.go:282] Add success. I1122 01:49:26.442271 3798194 net.go:642] Add success. I1122 01:49:26.443828 3798194 net.go:778] primary dev: ETH0 I1122 01:49:26.443844 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:49:26.443856 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:49:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:49:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:49:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:49:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:49:36.411983 3798194 memory.go:184] no items to output this cycle I1122 01:49:36.412143 3798194 cpu.go:275] no items to output this cycle E1122 01:49:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:49:46.411980 3798194 memory.go:184] no items to output this cycle I1122 01:49:46.412142 3798194 cpu.go:275] no items to output this cycle E1122 01:49:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:49:56.412262 3798194 memory.go:191] Add success. W1122 01:49:56.412288 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:49:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:49:56.412303 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:49:56.416378 3798194 cpu.go:282] Add success. I1122 01:49:56.441774 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:49:56.442286 3798194 net.go:642] Add success. I1122 01:49:56.443731 3798194 net.go:778] primary dev: ETH0 I1122 01:49:56.443746 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:49:56.443758 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:49:57.450654 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:49:57.450850 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:49:57.450864 3798194 disk_worker.go:723] disk space is not compliant W1122 01:49:57.450867 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:49:57.459353 3798194 disk_worker.go:509] system disk:vda1 I1122 01:49:57.459397 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:49:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:49:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:49:59.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:49:59.450128 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:49:59.471682 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:50:01.189897 3798194 disk_info.go:125] begin check local disk info of client I1122 01:50:01.193531 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:50:01.193540 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003dab80 0xc0003dabc0] E1122 01:50:06.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:50:06.411976 3798194 memory.go:184] no items to output this cycle I1122 01:50:06.412139 3798194 cpu.go:275] no items to output this cycle E1122 01:50:16.417279 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:50:16.417310 3798194 memory.go:184] no items to output this cycle I1122 01:50:16.417550 3798194 cpu.go:275] no items to output this cycle E1122 01:50:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:50:26.411958 3798194 memory.go:191] Add success. I1122 01:50:26.412401 3798194 cpu.go:282] Add success. I1122 01:50:26.441321 3798194 net.go:642] Add success. I1122 01:50:26.445414 3798194 net.go:778] primary dev: ETH0 I1122 01:50:26.445429 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:50:26.445443 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:50:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:50:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:50:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:50:36.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:50:36.411973 3798194 memory.go:184] no items to output this cycle I1122 01:50:36.412129 3798194 cpu.go:275] no items to output this cycle E1122 01:50:46.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:50:46.411984 3798194 memory.go:184] no items to output this cycle I1122 01:50:46.412136 3798194 cpu.go:275] no items to output this cycle E1122 01:50:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:50:56.411952 3798194 memory.go:191] Add success. W1122 01:50:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:50:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:50:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:50:56.412359 3798194 cpu.go:282] Add success. I1122 01:50:56.444379 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:50:56.444855 3798194 net.go:642] Add success. I1122 01:50:56.448705 3798194 net.go:778] primary dev: ETH0 I1122 01:50:56.448721 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:50:56.448733 3798194 net.go:659] Add eni metrics success, count: 7 E1122 01:50:57.450408 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 01:50:57.450422 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:50:57.450623 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:50:57.450632 3798194 disk_worker.go:723] disk space is not compliant W1122 01:50:57.450636 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:50:57.458667 3798194 disk_worker.go:509] system disk:vda1 I1122 01:50:57.458714 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 01:50:58.448682 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 01:50:58.448696 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 01:50:59.448263 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 01:50:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:50:59.450444 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:50:59.450462 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:50:59.470964 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:51:01.193646 3798194 disk_info.go:125] begin check local disk info of client I1122 01:51:01.197591 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:51:01.197600 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030c9c0 0xc00030ca00] E1122 01:51:06.412095 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:51:06.412135 3798194 memory.go:184] no items to output this cycle I1122 01:51:06.412281 3798194 cpu.go:275] no items to output this cycle E1122 01:51:16.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:51:16.412258 3798194 memory.go:184] no items to output this cycle I1122 01:51:16.412401 3798194 cpu.go:275] no items to output this cycle I1122 01:51:17.588278 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:51:17.588287 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:51:26.411341 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:51:26.411374 3798194 memory.go:191] Add success. I1122 01:51:26.411707 3798194 cpu.go:282] Add success. I1122 01:51:26.441276 3798194 net.go:642] Add success. I1122 01:51:26.445127 3798194 net.go:778] primary dev: ETH0 I1122 01:51:26.445143 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:51:26.445156 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:51:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:51:29.450086 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:51:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:51:36.412343 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:51:36.412366 3798194 memory.go:184] no items to output this cycle I1122 01:51:36.412511 3798194 cpu.go:275] no items to output this cycle E1122 01:51:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:51:46.411971 3798194 memory.go:184] no items to output this cycle I1122 01:51:46.412135 3798194 cpu.go:275] no items to output this cycle E1122 01:51:56.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:51:56.411946 3798194 memory.go:191] Add success. W1122 01:51:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:51:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:51:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:51:56.412330 3798194 cpu.go:282] Add success. I1122 01:51:56.440962 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:51:56.441480 3798194 net.go:642] Add success. I1122 01:51:56.445678 3798194 net.go:778] primary dev: ETH0 I1122 01:51:56.445693 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:51:56.445706 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:51:57.450333 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:51:57.450547 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:51:57.450558 3798194 disk_worker.go:723] disk space is not compliant W1122 01:51:57.450561 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:51:57.458622 3798194 disk_worker.go:509] system disk:vda1 I1122 01:51:57.458664 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:51:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:51:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:51:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:51:59.450140 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:51:59.471659 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:52:01.197710 3798194 disk_info.go:125] begin check local disk info of client I1122 01:52:01.201598 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:52:01.201607 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e100 0xc00030e140] E1122 01:52:06.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:52:06.413254 3798194 memory.go:184] no items to output this cycle I1122 01:52:06.413401 3798194 cpu.go:275] no items to output this cycle E1122 01:52:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:52:16.411988 3798194 memory.go:184] no items to output this cycle I1122 01:52:16.412152 3798194 cpu.go:275] no items to output this cycle E1122 01:52:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:52:26.411955 3798194 memory.go:191] Add success. I1122 01:52:26.412311 3798194 cpu.go:282] Add success. I1122 01:52:26.441913 3798194 net.go:642] Add success. I1122 01:52:26.446249 3798194 net.go:778] primary dev: ETH0 I1122 01:52:26.446264 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:52:26.446276 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:52:29.450008 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:52:29.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:52:29.450125 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:52:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:52:36.412256 3798194 memory.go:184] no items to output this cycle I1122 01:52:36.412398 3798194 cpu.go:275] no items to output this cycle E1122 01:52:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:52:46.411974 3798194 memory.go:184] no items to output this cycle I1122 01:52:46.412130 3798194 cpu.go:275] no items to output this cycle E1122 01:52:56.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:52:56.413259 3798194 memory.go:191] Add success. W1122 01:52:56.413293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:52:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:52:56.413309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:52:56.417305 3798194 cpu.go:282] Add success. I1122 01:52:56.441838 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:52:56.442427 3798194 net.go:642] Add success. I1122 01:52:56.444090 3798194 net.go:778] primary dev: ETH0 I1122 01:52:56.444105 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:52:56.444131 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:52:57.451325 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:52:57.451488 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:52:57.451543 3798194 disk_worker.go:723] disk space is not compliant W1122 01:52:57.451546 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:52:57.459897 3798194 disk_worker.go:509] system disk:vda1 I1122 01:52:57.459941 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:52:58.446369 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:52:59.450467 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:52:59.450547 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:52:59.450572 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:52:59.471096 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:53:01.204142 3798194 disk_info.go:125] begin check local disk info of client I1122 01:53:01.207706 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:53:01.207715 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047cd40 0xc00047cd80] E1122 01:53:06.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:53:06.411974 3798194 memory.go:184] no items to output this cycle I1122 01:53:06.412122 3798194 cpu.go:275] no items to output this cycle E1122 01:53:16.413256 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:53:16.413283 3798194 memory.go:184] no items to output this cycle I1122 01:53:16.413434 3798194 cpu.go:275] no items to output this cycle E1122 01:53:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:53:26.411954 3798194 memory.go:191] Add success. I1122 01:53:26.412334 3798194 cpu.go:282] Add success. I1122 01:53:26.442059 3798194 net.go:642] Add success. I1122 01:53:26.446476 3798194 net.go:778] primary dev: ETH0 I1122 01:53:26.446491 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:53:26.446504 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:53:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:53:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:53:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:53:36.413162 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:53:36.413185 3798194 memory.go:184] no items to output this cycle I1122 01:53:36.413334 3798194 cpu.go:275] no items to output this cycle E1122 01:53:46.413076 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:53:46.413099 3798194 memory.go:184] no items to output this cycle I1122 01:53:46.413263 3798194 cpu.go:275] no items to output this cycle E1122 01:53:56.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:53:56.413278 3798194 memory.go:191] Add success. W1122 01:53:56.413309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:53:56.413320 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:53:56.413324 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:53:56.417224 3798194 cpu.go:282] Add success. I1122 01:53:56.441800 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:53:56.442381 3798194 net.go:642] Add success. I1122 01:53:56.443835 3798194 net.go:778] primary dev: ETH0 I1122 01:53:56.443850 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:53:56.443862 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:53:57.450860 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:53:57.451021 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:53:57.451076 3798194 disk_worker.go:723] disk space is not compliant W1122 01:53:57.451079 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:53:57.459329 3798194 disk_worker.go:509] system disk:vda1 I1122 01:53:57.459373 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:53:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:53:59.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:53:59.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:53:59.450501 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:53:59.471002 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:54:01.207818 3798194 disk_info.go:125] begin check local disk info of client I1122 01:54:01.211493 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:54:01.211502 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a0380 0xc0003a03c0] E1122 01:54:06.411990 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:54:06.412013 3798194 memory.go:184] no items to output this cycle I1122 01:54:06.412176 3798194 cpu.go:275] no items to output this cycle E1122 01:54:16.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:54:16.411987 3798194 memory.go:184] no items to output this cycle I1122 01:54:16.412146 3798194 cpu.go:275] no items to output this cycle I1122 01:54:17.588448 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:54:17.588458 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:54:26.411220 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:54:26.411252 3798194 memory.go:191] Add success. I1122 01:54:26.411579 3798194 cpu.go:282] Add success. I1122 01:54:26.442230 3798194 net.go:642] Add success. I1122 01:54:26.446674 3798194 net.go:778] primary dev: ETH0 I1122 01:54:26.446689 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:54:26.446701 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:54:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:54:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:54:29.450145 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:54:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:54:36.412254 3798194 memory.go:184] no items to output this cycle I1122 01:54:36.412394 3798194 cpu.go:275] no items to output this cycle E1122 01:54:46.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:54:46.411970 3798194 memory.go:184] no items to output this cycle I1122 01:54:46.412328 3798194 cpu.go:275] no items to output this cycle E1122 01:54:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:54:56.411952 3798194 memory.go:191] Add success. W1122 01:54:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:54:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:54:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:54:56.412374 3798194 cpu.go:282] Add success. I1122 01:54:56.441821 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:54:56.442413 3798194 net.go:642] Add success. I1122 01:54:56.446543 3798194 net.go:778] primary dev: ETH0 I1122 01:54:56.446558 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:54:56.446571 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:54:57.451008 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:54:57.451252 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:54:57.451265 3798194 disk_worker.go:723] disk space is not compliant W1122 01:54:57.451268 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:54:57.464498 3798194 disk_worker.go:509] system disk:vda1 I1122 01:54:57.464540 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:54:58.445948 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:54:59.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:54:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:54:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:54:59.471643 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:55:01.211609 3798194 disk_info.go:125] begin check local disk info of client I1122 01:55:01.215216 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:55:01.215225 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000446500 0xc000446540] E1122 01:55:06.412760 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:55:06.412781 3798194 memory.go:184] no items to output this cycle I1122 01:55:06.412934 3798194 cpu.go:275] no items to output this cycle E1122 01:55:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:55:16.412265 3798194 memory.go:184] no items to output this cycle I1122 01:55:16.412410 3798194 cpu.go:275] no items to output this cycle E1122 01:55:26.411936 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:55:26.411980 3798194 memory.go:191] Add success. I1122 01:55:26.412361 3798194 cpu.go:282] Add success. I1122 01:55:26.441960 3798194 net.go:642] Add success. I1122 01:55:26.446001 3798194 net.go:778] primary dev: ETH0 I1122 01:55:26.446026 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:55:26.446039 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:55:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:55:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:55:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:55:36.413171 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:55:36.413196 3798194 memory.go:184] no items to output this cycle I1122 01:55:36.413338 3798194 cpu.go:275] no items to output this cycle E1122 01:55:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:55:46.411974 3798194 memory.go:184] no items to output this cycle I1122 01:55:46.412133 3798194 cpu.go:275] no items to output this cycle E1122 01:55:56.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:55:56.411957 3798194 memory.go:191] Add success. W1122 01:55:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:55:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:55:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:55:56.412327 3798194 cpu.go:282] Add success. I1122 01:55:56.441529 3798194 event_worker.go:152] Polling the log file for events... I1122 01:55:56.444457 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:55:56.444994 3798194 net.go:642] Add success. I1122 01:55:56.448986 3798194 net.go:778] primary dev: ETH0 I1122 01:55:56.449001 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:55:56.449013 3798194 net.go:659] Add eni metrics success, count: 7 E1122 01:55:57.450620 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 01:55:57.450635 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:55:57.450823 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:55:57.450834 3798194 disk_worker.go:723] disk space is not compliant W1122 01:55:57.450837 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:55:57.459179 3798194 disk_worker.go:509] system disk:vda1 I1122 01:55:57.459221 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 01:55:58.448097 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 01:55:58.448134 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 01:55:59.448590 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 01:55:59.449665 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:55:59.449715 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:55:59.449736 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:55:59.471240 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:56:01.218807 3798194 disk_info.go:125] begin check local disk info of client I1122 01:56:01.222418 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:56:01.222428 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004826c0 0xc000482700] E1122 01:56:06.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:56:06.412264 3798194 memory.go:184] no items to output this cycle I1122 01:56:06.412406 3798194 cpu.go:275] no items to output this cycle E1122 01:56:16.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:56:16.411992 3798194 memory.go:184] no items to output this cycle I1122 01:56:16.412159 3798194 cpu.go:275] no items to output this cycle E1122 01:56:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:56:26.413269 3798194 memory.go:191] Add success. I1122 01:56:26.417518 3798194 cpu.go:282] Add success. I1122 01:56:26.441249 3798194 net.go:642] Add success. I1122 01:56:26.442764 3798194 net.go:778] primary dev: ETH0 I1122 01:56:26.442780 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:56:26.442793 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:56:29.450632 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:56:29.450720 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:56:29.450746 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:56:36.411613 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:56:36.411637 3798194 memory.go:184] no items to output this cycle I1122 01:56:36.411780 3798194 cpu.go:275] no items to output this cycle E1122 01:56:46.412249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:56:46.412286 3798194 memory.go:184] no items to output this cycle I1122 01:56:46.412439 3798194 cpu.go:275] no items to output this cycle E1122 01:56:56.412845 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:56:56.412879 3798194 memory.go:191] Add success. W1122 01:56:56.412916 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:56:56.412928 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:56:56.412932 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:56:56.417197 3798194 cpu.go:282] Add success. I1122 01:56:56.441736 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:56:56.442247 3798194 net.go:642] Add success. I1122 01:56:56.443604 3798194 net.go:778] primary dev: ETH0 I1122 01:56:56.443619 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:56:56.443633 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:56:57.451312 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:56:57.451515 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:56:57.451526 3798194 disk_worker.go:723] disk space is not compliant W1122 01:56:57.451529 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:56:57.459929 3798194 disk_worker.go:509] system disk:vda1 I1122 01:56:57.459971 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:56:58.446162 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:56:59.450191 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:56:59.450274 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:56:59.450300 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:56:59.470837 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:57:01.222523 3798194 disk_info.go:125] begin check local disk info of client I1122 01:57:01.226057 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:57:01.226066 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00038f940 0xc00038f980] E1122 01:57:06.412786 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:57:06.412808 3798194 memory.go:184] no items to output this cycle I1122 01:57:06.412955 3798194 cpu.go:275] no items to output this cycle E1122 01:57:16.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:57:16.411994 3798194 memory.go:184] no items to output this cycle I1122 01:57:16.412159 3798194 cpu.go:275] no items to output this cycle I1122 01:57:17.588620 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 01:57:17.588631 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 01:57:26.411388 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:57:26.411423 3798194 memory.go:191] Add success. I1122 01:57:26.411788 3798194 cpu.go:282] Add success. I1122 01:57:26.441318 3798194 net.go:642] Add success. I1122 01:57:26.445382 3798194 net.go:778] primary dev: ETH0 I1122 01:57:26.445400 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:57:26.445414 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:57:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:57:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:57:29.450627 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:57:36.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:57:36.413256 3798194 memory.go:184] no items to output this cycle I1122 01:57:36.413408 3798194 cpu.go:275] no items to output this cycle E1122 01:57:46.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:57:46.412260 3798194 memory.go:184] no items to output this cycle I1122 01:57:46.412402 3798194 cpu.go:275] no items to output this cycle E1122 01:57:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:57:56.411959 3798194 memory.go:191] Add success. W1122 01:57:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:57:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:57:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:57:56.412337 3798194 cpu.go:282] Add success. I1122 01:57:56.441825 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:57:56.442336 3798194 net.go:642] Add success. I1122 01:57:56.446572 3798194 net.go:778] primary dev: ETH0 I1122 01:57:56.446587 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:57:56.446598 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:57:57.450761 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:57:57.450934 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:57:57.450991 3798194 disk_worker.go:723] disk space is not compliant W1122 01:57:57.450995 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:57:57.463972 3798194 disk_worker.go:509] system disk:vda1 I1122 01:57:57.464014 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:57:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:57:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:57:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:57:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:57:59.471692 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:58:01.229861 3798194 disk_info.go:125] begin check local disk info of client I1122 01:58:01.233433 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:58:01.233443 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e6f00 0xc0003e6f40] E1122 01:58:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:58:06.412257 3798194 memory.go:184] no items to output this cycle I1122 01:58:06.412400 3798194 cpu.go:275] no items to output this cycle E1122 01:58:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:58:16.411984 3798194 memory.go:184] no items to output this cycle I1122 01:58:16.412145 3798194 cpu.go:275] no items to output this cycle E1122 01:58:26.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:58:26.411947 3798194 memory.go:191] Add success. I1122 01:58:26.412291 3798194 cpu.go:282] Add success. I1122 01:58:26.441758 3798194 net.go:642] Add success. I1122 01:58:26.445678 3798194 net.go:778] primary dev: ETH0 I1122 01:58:26.445696 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:58:26.445709 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:58:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:58:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:58:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:58:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:58:36.412255 3798194 memory.go:184] no items to output this cycle I1122 01:58:36.412401 3798194 cpu.go:275] no items to output this cycle E1122 01:58:46.413254 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:58:46.413281 3798194 memory.go:184] no items to output this cycle I1122 01:58:46.413431 3798194 cpu.go:275] no items to output this cycle E1122 01:58:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:58:56.411948 3798194 memory.go:191] Add success. W1122 01:58:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:58:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:58:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:58:56.412363 3798194 cpu.go:282] Add success. I1122 01:58:56.441835 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:58:56.442341 3798194 net.go:642] Add success. I1122 01:58:56.447186 3798194 net.go:778] primary dev: ETH0 I1122 01:58:56.447201 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:58:56.447215 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:58:57.461347 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:58:57.461611 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:58:57.461623 3798194 disk_worker.go:723] disk space is not compliant W1122 01:58:57.461626 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:58:57.465292 3798194 disk_worker.go:509] system disk:vda1 I1122 01:58:57.465330 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:58:58.446021 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:58:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:58:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:58:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:58:59.471658 3798194 disk_local_worker.go:436] Get disk info: [] I1122 01:59:01.233558 3798194 disk_info.go:125] begin check local disk info of client I1122 01:59:01.237424 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 01:59:01.237432 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004afb00 0xc0004afb40] E1122 01:59:06.412192 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:59:06.412214 3798194 memory.go:184] no items to output this cycle I1122 01:59:06.412361 3798194 cpu.go:275] no items to output this cycle E1122 01:59:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:59:16.411983 3798194 memory.go:184] no items to output this cycle I1122 01:59:16.412138 3798194 cpu.go:275] no items to output this cycle E1122 01:59:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:59:26.411954 3798194 memory.go:191] Add success. I1122 01:59:26.412314 3798194 cpu.go:282] Add success. I1122 01:59:26.441921 3798194 net.go:642] Add success. I1122 01:59:26.446396 3798194 net.go:778] primary dev: ETH0 I1122 01:59:26.446411 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:59:26.446426 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:59:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:59:29.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:59:29.450489 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 01:59:36.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:59:36.413254 3798194 memory.go:184] no items to output this cycle I1122 01:59:36.413403 3798194 cpu.go:275] no items to output this cycle E1122 01:59:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:59:46.411978 3798194 memory.go:184] no items to output this cycle I1122 01:59:46.412141 3798194 cpu.go:275] no items to output this cycle E1122 01:59:56.412819 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 01:59:56.412865 3798194 memory.go:191] Add success. W1122 01:59:56.412901 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 01:59:56.412918 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 01:59:56.412923 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 01:59:56.423541 3798194 cpu.go:282] Add success. I1122 01:59:56.441706 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 01:59:56.442512 3798194 net.go:642] Add success. I1122 01:59:56.444322 3798194 net.go:778] primary dev: ETH0 I1122 01:59:56.444347 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 01:59:56.444365 3798194 net.go:659] Add eni metrics success, count: 7 I1122 01:59:57.445166 3798194 custom_config.go:106] query custom config with name: gpu W1122 01:59:57.445424 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 01:59:57.445436 3798194 disk_worker.go:723] disk space is not compliant W1122 01:59:57.445440 3798194 disk_worker.go:743] disk inode is not compliant I1122 01:59:57.447553 3798194 disk_worker.go:509] system disk:vda1 I1122 01:59:57.447595 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 01:59:58.445988 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 01:59:59.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 01:59:59.450127 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 01:59:59.450154 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 01:59:59.471650 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:00:01.237535 3798194 disk_info.go:125] begin check local disk info of client I1122 02:00:01.241439 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:00:01.241449 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000402fc0 0xc000403000] E1122 02:00:06.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:00:06.412877 3798194 memory.go:184] no items to output this cycle I1122 02:00:06.413022 3798194 cpu.go:275] no items to output this cycle E1122 02:00:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:00:16.411979 3798194 memory.go:184] no items to output this cycle I1122 02:00:16.412142 3798194 cpu.go:275] no items to output this cycle I1122 02:00:17.589273 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:00:17.589282 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:00:26.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:00:26.412889 3798194 memory.go:191] Add success. I1122 02:00:26.417496 3798194 cpu.go:282] Add success. I1122 02:00:26.442045 3798194 net.go:642] Add success. I1122 02:00:26.443585 3798194 net.go:778] primary dev: ETH0 I1122 02:00:26.443600 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:00:26.443613 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:00:29.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:00:29.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:00:29.450515 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:00:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:00:36.411978 3798194 memory.go:184] no items to output this cycle I1122 02:00:36.412139 3798194 cpu.go:275] no items to output this cycle E1122 02:00:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:00:46.411982 3798194 memory.go:184] no items to output this cycle I1122 02:00:46.412146 3798194 cpu.go:275] no items to output this cycle E1122 02:00:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:00:56.411945 3798194 memory.go:191] Add success. W1122 02:00:56.411971 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:00:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:00:56.411985 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:00:56.412342 3798194 cpu.go:282] Add success. I1122 02:00:56.444685 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:00:56.445298 3798194 net.go:642] Add success. I1122 02:00:56.449455 3798194 net.go:778] primary dev: ETH0 I1122 02:00:56.449471 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:00:56.449484 3798194 net.go:659] Add eni metrics success, count: 7 E1122 02:00:57.450314 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 02:00:57.450329 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:00:57.450524 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:00:57.450534 3798194 disk_worker.go:723] disk space is not compliant W1122 02:00:57.450537 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:00:57.458670 3798194 disk_worker.go:509] system disk:vda1 I1122 02:00:57.458708 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 02:00:58.447808 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 02:00:58.447823 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 02:00:59.448582 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 02:00:59.449657 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:00:59.449707 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:00:59.449724 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:00:59.471189 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:01:01.244930 3798194 disk_info.go:125] begin check local disk info of client I1122 02:01:01.248806 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:01:01.248815 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003135c0 0xc000313600] E1122 02:01:06.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:01:06.413253 3798194 memory.go:184] no items to output this cycle I1122 02:01:06.413397 3798194 cpu.go:275] no items to output this cycle E1122 02:01:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:01:16.411984 3798194 memory.go:184] no items to output this cycle I1122 02:01:16.412153 3798194 cpu.go:275] no items to output this cycle E1122 02:01:26.412483 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:01:26.412515 3798194 memory.go:191] Add success. I1122 02:01:26.413720 3798194 cpu.go:282] Add success. I1122 02:01:26.441445 3798194 net.go:642] Add success. I1122 02:01:26.442907 3798194 net.go:778] primary dev: ETH0 I1122 02:01:26.442922 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:01:26.442935 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:01:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:01:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:01:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:01:36.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:01:36.411971 3798194 memory.go:184] no items to output this cycle I1122 02:01:36.412107 3798194 cpu.go:275] no items to output this cycle E1122 02:01:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:01:46.411984 3798194 memory.go:184] no items to output this cycle I1122 02:01:46.412144 3798194 cpu.go:275] no items to output this cycle E1122 02:01:56.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:01:56.412895 3798194 memory.go:191] Add success. W1122 02:01:56.412922 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:01:56.412933 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:01:56.412936 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:01:56.417251 3798194 cpu.go:282] Add success. I1122 02:01:56.441651 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:01:56.442154 3798194 net.go:642] Add success. I1122 02:01:56.443669 3798194 net.go:778] primary dev: ETH0 I1122 02:01:56.443683 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:01:56.443695 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:01:57.450660 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:01:57.450854 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:01:57.450866 3798194 disk_worker.go:723] disk space is not compliant W1122 02:01:57.450869 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:01:57.459009 3798194 disk_worker.go:509] system disk:vda1 I1122 02:01:57.459055 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:01:58.446386 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:01:59.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:01:59.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:01:59.450504 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:01:59.471011 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:02:01.248918 3798194 disk_info.go:125] begin check local disk info of client I1122 02:02:01.252750 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:02:01.252760 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0f00 0xc0002e0f40] E1122 02:02:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:02:06.411985 3798194 memory.go:184] no items to output this cycle I1122 02:02:06.412142 3798194 cpu.go:275] no items to output this cycle E1122 02:02:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:02:16.411974 3798194 memory.go:184] no items to output this cycle I1122 02:02:16.412134 3798194 cpu.go:275] no items to output this cycle E1122 02:02:26.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:02:26.412889 3798194 memory.go:191] Add success. I1122 02:02:26.416839 3798194 cpu.go:282] Add success. I1122 02:02:26.441292 3798194 net.go:642] Add success. I1122 02:02:26.442898 3798194 net.go:778] primary dev: ETH0 I1122 02:02:26.442914 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:02:26.442926 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:02:29.450633 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:02:29.450710 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:02:29.450740 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:02:36.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:02:36.412262 3798194 memory.go:184] no items to output this cycle I1122 02:02:36.412407 3798194 cpu.go:275] no items to output this cycle E1122 02:02:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:02:46.411974 3798194 memory.go:184] no items to output this cycle I1122 02:02:46.412150 3798194 cpu.go:275] no items to output this cycle E1122 02:02:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:02:56.411947 3798194 memory.go:191] Add success. W1122 02:02:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:02:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:02:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:02:56.412354 3798194 cpu.go:282] Add success. I1122 02:02:56.441839 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:02:56.442444 3798194 net.go:642] Add success. I1122 02:02:56.446640 3798194 net.go:778] primary dev: ETH0 I1122 02:02:56.446657 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:02:56.446675 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:02:57.450655 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:02:57.450861 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:02:57.450872 3798194 disk_worker.go:723] disk space is not compliant W1122 02:02:57.450875 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:02:57.459270 3798194 disk_worker.go:509] system disk:vda1 I1122 02:02:57.459312 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:02:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:02:59.450121 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:02:59.450195 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:02:59.450228 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:02:59.470700 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:03:01.256137 3798194 disk_info.go:125] begin check local disk info of client I1122 02:03:01.259540 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:03:01.259549 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047c100 0xc00047c140] E1122 02:03:06.412359 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:03:06.412458 3798194 memory.go:184] no items to output this cycle I1122 02:03:06.412883 3798194 cpu.go:275] no items to output this cycle E1122 02:03:16.411942 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:03:16.411964 3798194 memory.go:184] no items to output this cycle I1122 02:03:16.412126 3798194 cpu.go:275] no items to output this cycle I1122 02:03:17.589440 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:03:17.589448 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:03:26.411460 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:03:26.411492 3798194 memory.go:191] Add success. I1122 02:03:26.411843 3798194 cpu.go:282] Add success. I1122 02:03:26.441454 3798194 net.go:642] Add success. I1122 02:03:26.445678 3798194 net.go:778] primary dev: ETH0 I1122 02:03:26.445693 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:03:26.445708 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:03:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:03:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:03:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:03:36.412485 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:03:36.412507 3798194 memory.go:184] no items to output this cycle I1122 02:03:36.412657 3798194 cpu.go:275] no items to output this cycle E1122 02:03:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:03:46.413259 3798194 memory.go:184] no items to output this cycle I1122 02:03:46.413408 3798194 cpu.go:275] no items to output this cycle E1122 02:03:56.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:03:56.411959 3798194 memory.go:191] Add success. W1122 02:03:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:03:56.412001 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:03:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:03:56.412338 3798194 cpu.go:282] Add success. I1122 02:03:56.441800 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:03:56.442382 3798194 net.go:642] Add success. I1122 02:03:56.446624 3798194 net.go:778] primary dev: ETH0 I1122 02:03:56.446639 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:03:56.446653 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:03:57.450286 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:03:57.450463 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:03:57.450520 3798194 disk_worker.go:723] disk space is not compliant W1122 02:03:57.450524 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:03:57.458564 3798194 disk_worker.go:509] system disk:vda1 I1122 02:03:57.458603 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:03:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:03:59.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:03:59.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:03:59.450495 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:03:59.470994 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:04:01.259652 3798194 disk_info.go:125] begin check local disk info of client I1122 02:04:01.263211 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:04:01.263224 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037d340 0xc00037d380] E1122 02:04:06.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:04:06.412253 3798194 memory.go:184] no items to output this cycle I1122 02:04:06.412395 3798194 cpu.go:275] no items to output this cycle E1122 02:04:16.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:04:16.411996 3798194 memory.go:184] no items to output this cycle I1122 02:04:16.412157 3798194 cpu.go:275] no items to output this cycle E1122 02:04:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:04:26.411952 3798194 memory.go:191] Add success. I1122 02:04:26.412318 3798194 cpu.go:282] Add success. I1122 02:04:26.441928 3798194 net.go:642] Add success. I1122 02:04:26.446364 3798194 net.go:778] primary dev: ETH0 I1122 02:04:26.446381 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:04:26.446393 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:04:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:04:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:04:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:04:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:04:36.412254 3798194 memory.go:184] no items to output this cycle I1122 02:04:36.412395 3798194 cpu.go:275] no items to output this cycle E1122 02:04:46.412846 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:04:46.412870 3798194 memory.go:184] no items to output this cycle I1122 02:04:46.413019 3798194 cpu.go:275] no items to output this cycle E1122 02:04:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:04:56.412270 3798194 memory.go:191] Add success. W1122 02:04:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:04:56.412315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:04:56.412318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:04:56.416359 3798194 cpu.go:282] Add success. I1122 02:04:56.441730 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:04:56.442322 3798194 net.go:642] Add success. I1122 02:04:56.443843 3798194 net.go:778] primary dev: ETH0 I1122 02:04:56.443858 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:04:56.443882 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:04:57.452227 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:04:57.452396 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:04:57.452454 3798194 disk_worker.go:723] disk space is not compliant W1122 02:04:57.452457 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:04:57.460595 3798194 disk_worker.go:509] system disk:vda1 I1122 02:04:57.460641 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:04:58.446398 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:04:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:04:59.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:04:59.450123 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:04:59.471649 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:05:01.266147 3798194 disk_info.go:125] begin check local disk info of client I1122 02:05:01.270014 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:05:01.270023 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00043bcc0 0xc00043bd00] E1122 02:05:06.413039 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:05:06.413065 3798194 memory.go:184] no items to output this cycle I1122 02:05:06.413232 3798194 cpu.go:275] no items to output this cycle E1122 02:05:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:05:16.413257 3798194 memory.go:184] no items to output this cycle I1122 02:05:16.413405 3798194 cpu.go:275] no items to output this cycle E1122 02:05:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:05:26.411953 3798194 memory.go:191] Add success. I1122 02:05:26.416011 3798194 cpu.go:282] Add success. I1122 02:05:26.441542 3798194 net.go:642] Add success. I1122 02:05:26.447362 3798194 net.go:778] primary dev: ETH0 I1122 02:05:26.447378 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:05:26.447392 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:05:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:05:29.450465 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:05:29.450490 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:05:36.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:05:36.413258 3798194 memory.go:184] no items to output this cycle I1122 02:05:36.413404 3798194 cpu.go:275] no items to output this cycle E1122 02:05:46.411973 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:05:46.411997 3798194 memory.go:184] no items to output this cycle I1122 02:05:46.412160 3798194 cpu.go:275] no items to output this cycle E1122 02:05:56.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:05:56.412890 3798194 memory.go:191] Add success. W1122 02:05:56.412922 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:05:56.412933 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:05:56.412936 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:05:56.417279 3798194 cpu.go:282] Add success. I1122 02:05:56.441427 3798194 event_worker.go:152] Polling the log file for events... I1122 02:05:56.444719 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:05:56.445525 3798194 net.go:642] Add success. I1122 02:05:56.447008 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795282556, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 02:05:56.447265 3798194 net.go:778] primary dev: ETH0 I1122 02:05:56.447280 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:05:56.447310 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:05:56.460250 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 02:05:57.449270 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 02:05:57.449284 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:05:57.449518 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:05:57.449528 3798194 disk_worker.go:723] disk space is not compliant W1122 02:05:57.449531 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:05:57.451351 3798194 disk_worker.go:509] system disk:vda1 I1122 02:05:57.451391 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 02:05:58.448579 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 02:05:58.448595 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 02:05:59.448933 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 02:05:59.450007 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:05:59.450059 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:05:59.450077 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:05:59.471598 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:06:01.273579 3798194 disk_info.go:125] begin check local disk info of client I1122 02:06:01.278074 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:06:01.278083 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00049ad00 0xc00049ad40] E1122 02:06:06.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:06:06.411977 3798194 memory.go:184] no items to output this cycle I1122 02:06:06.412132 3798194 cpu.go:275] no items to output this cycle E1122 02:06:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:06:16.411971 3798194 memory.go:184] no items to output this cycle I1122 02:06:16.412133 3798194 cpu.go:275] no items to output this cycle I1122 02:06:17.589608 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:06:17.589617 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:06:26.411372 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:06:26.411405 3798194 memory.go:191] Add success. I1122 02:06:26.411749 3798194 cpu.go:282] Add success. I1122 02:06:26.441309 3798194 net.go:642] Add success. I1122 02:06:26.445244 3798194 net.go:778] primary dev: ETH0 I1122 02:06:26.445260 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:06:26.445273 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:06:29.450450 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:06:29.450518 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:06:29.450546 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:06:36.412401 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:06:36.412423 3798194 memory.go:184] no items to output this cycle I1122 02:06:36.412578 3798194 cpu.go:275] no items to output this cycle E1122 02:06:46.413287 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:06:46.413310 3798194 memory.go:184] no items to output this cycle I1122 02:06:46.413465 3798194 cpu.go:275] no items to output this cycle E1122 02:06:56.412867 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:06:56.412901 3798194 memory.go:191] Add success. W1122 02:06:56.412936 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:06:56.412947 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:06:56.412951 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:06:56.419464 3798194 cpu.go:282] Add success. I1122 02:06:56.440915 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:06:56.441472 3798194 net.go:642] Add success. I1122 02:06:56.443046 3798194 net.go:778] primary dev: ETH0 I1122 02:06:56.443060 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:06:56.443072 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:06:57.445249 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:06:57.445468 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:06:57.445547 3798194 disk_worker.go:723] disk space is not compliant W1122 02:06:57.445551 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:06:57.448127 3798194 disk_worker.go:509] system disk:vda1 I1122 02:06:57.448169 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:06:58.446373 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:06:59.450480 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:06:59.450554 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:06:59.450580 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:06:59.471090 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:07:01.281142 3798194 disk_info.go:125] begin check local disk info of client I1122 02:07:01.284918 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:07:01.284926 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000404c80 0xc000404cc0] E1122 02:07:06.412990 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:07:06.413014 3798194 memory.go:184] no items to output this cycle I1122 02:07:06.413321 3798194 cpu.go:275] no items to output this cycle E1122 02:07:16.412246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:07:16.412270 3798194 memory.go:184] no items to output this cycle I1122 02:07:16.412423 3798194 cpu.go:275] no items to output this cycle E1122 02:07:26.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:07:26.411957 3798194 memory.go:191] Add success. I1122 02:07:26.412329 3798194 cpu.go:282] Add success. I1122 02:07:26.441952 3798194 net.go:642] Add success. I1122 02:07:26.446045 3798194 net.go:778] primary dev: ETH0 I1122 02:07:26.446061 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:07:26.446078 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:07:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:07:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:07:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:07:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:07:36.412252 3798194 memory.go:184] no items to output this cycle I1122 02:07:36.412407 3798194 cpu.go:275] no items to output this cycle E1122 02:07:46.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:07:46.411972 3798194 memory.go:184] no items to output this cycle I1122 02:07:46.412131 3798194 cpu.go:275] no items to output this cycle E1122 02:07:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:07:56.411948 3798194 memory.go:191] Add success. W1122 02:07:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:07:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:07:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:07:56.412328 3798194 cpu.go:282] Add success. I1122 02:07:56.441806 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:07:56.442412 3798194 net.go:642] Add success. I1122 02:07:56.446641 3798194 net.go:778] primary dev: ETH0 I1122 02:07:56.446657 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:07:56.446669 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:07:57.457039 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:07:57.457278 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:07:57.457290 3798194 disk_worker.go:723] disk space is not compliant W1122 02:07:57.457293 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:07:57.475046 3798194 disk_worker.go:509] system disk:vda1 I1122 02:07:57.475105 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:07:58.445985 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:07:59.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:07:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:07:59.450155 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:07:59.471646 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:08:01.288522 3798194 disk_info.go:125] begin check local disk info of client I1122 02:08:01.292425 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:08:01.292434 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000347400 0xc000347440] E1122 02:08:06.412756 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:08:06.412777 3798194 memory.go:184] no items to output this cycle I1122 02:08:06.412927 3798194 cpu.go:275] no items to output this cycle E1122 02:08:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:08:16.411980 3798194 memory.go:184] no items to output this cycle I1122 02:08:16.412145 3798194 cpu.go:275] no items to output this cycle E1122 02:08:26.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:08:26.411945 3798194 memory.go:191] Add success. I1122 02:08:26.412302 3798194 cpu.go:282] Add success. I1122 02:08:26.441915 3798194 net.go:642] Add success. I1122 02:08:26.446316 3798194 net.go:778] primary dev: ETH0 I1122 02:08:26.446332 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:08:26.446470 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:08:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:08:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:08:29.450147 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:08:36.412088 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:08:36.412125 3798194 memory.go:184] no items to output this cycle I1122 02:08:36.412271 3798194 cpu.go:275] no items to output this cycle E1122 02:08:46.412335 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:08:46.412357 3798194 memory.go:184] no items to output this cycle I1122 02:08:46.412504 3798194 cpu.go:275] no items to output this cycle E1122 02:08:56.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:08:56.413264 3798194 memory.go:191] Add success. W1122 02:08:56.413291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:08:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:08:56.413306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:08:56.417516 3798194 cpu.go:282] Add success. I1122 02:08:56.441892 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:08:56.442390 3798194 net.go:642] Add success. I1122 02:08:56.443897 3798194 net.go:778] primary dev: ETH0 I1122 02:08:56.443910 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:08:56.443923 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:08:57.451273 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:08:57.451477 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:08:57.451488 3798194 disk_worker.go:723] disk space is not compliant W1122 02:08:57.451491 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:08:57.459912 3798194 disk_worker.go:509] system disk:vda1 I1122 02:08:57.459951 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:08:58.446503 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:08:59.450607 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:08:59.450688 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:08:59.450713 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:08:59.471190 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:09:01.292538 3798194 disk_info.go:125] begin check local disk info of client I1122 02:09:01.296172 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:09:01.296181 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a96c0 0xc0003a9700] E1122 02:09:06.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:09:06.411980 3798194 memory.go:184] no items to output this cycle I1122 02:09:06.412142 3798194 cpu.go:275] no items to output this cycle E1122 02:09:16.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:09:16.412879 3798194 memory.go:184] no items to output this cycle I1122 02:09:16.413022 3798194 cpu.go:275] no items to output this cycle I1122 02:09:17.590102 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:09:17.590121 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:09:26.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:09:26.412891 3798194 memory.go:191] Add success. I1122 02:09:26.417155 3798194 cpu.go:282] Add success. I1122 02:09:26.441637 3798194 net.go:642] Add success. I1122 02:09:26.443022 3798194 net.go:778] primary dev: ETH0 I1122 02:09:26.443038 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:09:26.443051 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:09:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:09:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:09:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:09:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:09:36.411975 3798194 memory.go:184] no items to output this cycle I1122 02:09:36.412129 3798194 cpu.go:275] no items to output this cycle E1122 02:09:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:09:46.411976 3798194 memory.go:184] no items to output this cycle I1122 02:09:46.412128 3798194 cpu.go:275] no items to output this cycle E1122 02:09:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:09:56.412266 3798194 memory.go:191] Add success. W1122 02:09:56.412292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:09:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:09:56.412306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:09:56.416528 3798194 cpu.go:282] Add success. I1122 02:09:56.442045 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:09:56.442889 3798194 net.go:642] Add success. I1122 02:09:56.444388 3798194 net.go:778] primary dev: ETH0 I1122 02:09:56.444404 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:09:56.444417 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:09:57.450363 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:09:57.450564 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:09:57.450575 3798194 disk_worker.go:723] disk space is not compliant W1122 02:09:57.450578 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:09:57.458666 3798194 disk_worker.go:509] system disk:vda1 I1122 02:09:57.458710 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:09:58.446392 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:09:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:09:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:09:59.450140 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:09:59.470882 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:10:01.300150 3798194 disk_info.go:125] begin check local disk info of client I1122 02:10:01.303891 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:10:01.303900 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000481140 0xc000481180] E1122 02:10:06.412688 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:10:06.412709 3798194 memory.go:184] no items to output this cycle I1122 02:10:06.412852 3798194 cpu.go:275] no items to output this cycle E1122 02:10:16.413319 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:10:16.413339 3798194 memory.go:184] no items to output this cycle I1122 02:10:16.413488 3798194 cpu.go:275] no items to output this cycle E1122 02:10:26.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:10:26.411961 3798194 memory.go:191] Add success. I1122 02:10:26.412334 3798194 cpu.go:282] Add success. I1122 02:10:26.441941 3798194 net.go:642] Add success. I1122 02:10:26.446266 3798194 net.go:778] primary dev: ETH0 I1122 02:10:26.446281 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:10:26.446294 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:10:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:10:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:10:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:10:36.412280 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:10:36.412311 3798194 memory.go:184] no items to output this cycle I1122 02:10:36.412510 3798194 cpu.go:275] no items to output this cycle E1122 02:10:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:10:46.413254 3798194 memory.go:184] no items to output this cycle I1122 02:10:46.413400 3798194 cpu.go:275] no items to output this cycle E1122 02:10:56.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:10:56.412263 3798194 memory.go:191] Add success. W1122 02:10:56.412288 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:10:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:10:56.412302 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:10:56.416422 3798194 cpu.go:282] Add success. I1122 02:10:56.444314 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:10:56.444800 3798194 net.go:642] Add success. I1122 02:10:56.446267 3798194 net.go:778] primary dev: ETH0 I1122 02:10:56.446282 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:10:56.446293 3798194 net.go:659] Add eni metrics success, count: 7 E1122 02:10:57.450264 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 02:10:57.450277 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:10:57.450439 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:10:57.450490 3798194 disk_worker.go:723] disk space is not compliant W1122 02:10:57.450493 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:10:57.458601 3798194 disk_worker.go:509] system disk:vda1 I1122 02:10:57.458644 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 02:10:58.447802 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 02:10:58.447816 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 02:10:59.448624 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 02:10:59.449699 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:10:59.449751 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:10:59.449770 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:10:59.471286 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:11:01.307134 3798194 disk_info.go:125] begin check local disk info of client I1122 02:11:01.310960 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:11:01.310969 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fea40 0xc0003fea80] E1122 02:11:06.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:11:06.411980 3798194 memory.go:184] no items to output this cycle I1122 02:11:06.412134 3798194 cpu.go:275] no items to output this cycle E1122 02:11:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:11:16.411981 3798194 memory.go:184] no items to output this cycle I1122 02:11:16.412140 3798194 cpu.go:275] no items to output this cycle E1122 02:11:26.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:11:26.412271 3798194 memory.go:191] Add success. I1122 02:11:26.416546 3798194 cpu.go:282] Add success. I1122 02:11:26.442086 3798194 net.go:642] Add success. I1122 02:11:26.443722 3798194 net.go:778] primary dev: ETH0 I1122 02:11:26.443736 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:11:26.443749 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:11:29.450384 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:11:29.450462 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:11:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:11:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:11:36.413255 3798194 memory.go:184] no items to output this cycle I1122 02:11:36.413405 3798194 cpu.go:275] no items to output this cycle E1122 02:11:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:11:46.411978 3798194 memory.go:184] no items to output this cycle I1122 02:11:46.412133 3798194 cpu.go:275] no items to output this cycle E1122 02:11:56.411932 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:11:56.411967 3798194 memory.go:191] Add success. W1122 02:11:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:11:56.412010 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:11:56.412015 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:11:56.412338 3798194 cpu.go:282] Add success. I1122 02:11:56.441814 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:11:56.442393 3798194 net.go:642] Add success. I1122 02:11:56.446816 3798194 net.go:778] primary dev: ETH0 I1122 02:11:56.446831 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:11:56.446845 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:11:57.450295 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:11:57.450469 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:11:57.450526 3798194 disk_worker.go:723] disk space is not compliant W1122 02:11:57.450530 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:11:57.458560 3798194 disk_worker.go:509] system disk:vda1 I1122 02:11:57.458600 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:11:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:11:59.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:11:59.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:11:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:11:59.471653 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:12:01.314667 3798194 disk_info.go:125] begin check local disk info of client I1122 02:12:01.318217 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:12:01.318226 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000372340 0xc000372380] E1122 02:12:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:12:06.412260 3798194 memory.go:184] no items to output this cycle I1122 02:12:06.412409 3798194 cpu.go:275] no items to output this cycle E1122 02:12:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:12:16.411975 3798194 memory.go:184] no items to output this cycle I1122 02:12:16.412133 3798194 cpu.go:275] no items to output this cycle I1122 02:12:17.590276 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:12:17.590284 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:12:26.411476 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:12:26.411509 3798194 memory.go:191] Add success. I1122 02:12:26.411842 3798194 cpu.go:282] Add success. I1122 02:12:26.441480 3798194 net.go:642] Add success. I1122 02:12:26.445742 3798194 net.go:778] primary dev: ETH0 I1122 02:12:26.445757 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:12:26.445769 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:12:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:12:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:12:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:12:36.412813 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:12:36.412838 3798194 memory.go:184] no items to output this cycle I1122 02:12:36.412986 3798194 cpu.go:275] no items to output this cycle E1122 02:12:46.412172 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:12:46.412195 3798194 memory.go:184] no items to output this cycle I1122 02:12:46.412514 3798194 cpu.go:275] no items to output this cycle E1122 02:12:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:12:56.411946 3798194 memory.go:191] Add success. W1122 02:12:56.411974 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:12:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:12:56.411988 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:12:56.412377 3798194 cpu.go:282] Add success. I1122 02:12:56.441942 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:12:56.442542 3798194 net.go:642] Add success. I1122 02:12:56.446934 3798194 net.go:778] primary dev: ETH0 I1122 02:12:56.446948 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:12:56.446962 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:12:57.450322 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:12:57.450524 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:12:57.450536 3798194 disk_worker.go:723] disk space is not compliant W1122 02:12:57.450539 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:12:57.458681 3798194 disk_worker.go:509] system disk:vda1 I1122 02:12:57.458723 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:12:58.445997 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:12:59.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:12:59.450490 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:12:59.450516 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:12:59.471037 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:13:01.322145 3798194 disk_info.go:125] begin check local disk info of client I1122 02:13:01.326087 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:13:01.326096 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00042e500 0xc00042e540] E1122 02:13:06.412091 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:13:06.412127 3798194 memory.go:184] no items to output this cycle I1122 02:13:06.412280 3798194 cpu.go:275] no items to output this cycle E1122 02:13:16.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:13:16.412884 3798194 memory.go:184] no items to output this cycle I1122 02:13:16.413037 3798194 cpu.go:275] no items to output this cycle E1122 02:13:26.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:13:26.413266 3798194 memory.go:191] Add success. I1122 02:13:26.417601 3798194 cpu.go:282] Add success. I1122 02:13:26.442222 3798194 net.go:642] Add success. I1122 02:13:26.443742 3798194 net.go:778] primary dev: ETH0 I1122 02:13:26.443757 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:13:26.443768 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:13:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:13:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:13:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:13:36.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:13:36.411988 3798194 memory.go:184] no items to output this cycle I1122 02:13:36.412141 3798194 cpu.go:275] no items to output this cycle E1122 02:13:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:13:46.411982 3798194 memory.go:184] no items to output this cycle I1122 02:13:46.412139 3798194 cpu.go:275] no items to output this cycle E1122 02:13:56.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:13:56.412272 3798194 memory.go:191] Add success. W1122 02:13:56.412296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:13:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:13:56.412311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:13:56.416495 3798194 cpu.go:282] Add success. I1122 02:13:56.441910 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:13:56.442487 3798194 net.go:642] Add success. I1122 02:13:56.444051 3798194 net.go:778] primary dev: ETH0 I1122 02:13:56.444066 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:13:56.444081 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:13:57.450728 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:13:57.450929 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:13:57.450940 3798194 disk_worker.go:723] disk space is not compliant W1122 02:13:57.450943 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:13:57.459381 3798194 disk_worker.go:509] system disk:vda1 I1122 02:13:57.459420 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:13:58.446196 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:13:59.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:13:59.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:13:59.450514 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:13:59.471058 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:14:01.330354 3798194 disk_info.go:125] begin check local disk info of client I1122 02:14:01.334268 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:14:01.334277 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000070a80 0xc000070ac0] E1122 02:14:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:14:06.411976 3798194 memory.go:184] no items to output this cycle I1122 02:14:06.412132 3798194 cpu.go:275] no items to output this cycle E1122 02:14:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:14:16.411977 3798194 memory.go:184] no items to output this cycle I1122 02:14:16.412135 3798194 cpu.go:275] no items to output this cycle E1122 02:14:26.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:14:26.412883 3798194 memory.go:191] Add success. I1122 02:14:26.416947 3798194 cpu.go:282] Add success. I1122 02:14:26.441508 3798194 net.go:642] Add success. I1122 02:14:26.442979 3798194 net.go:778] primary dev: ETH0 I1122 02:14:26.442994 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:14:26.443008 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:14:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:14:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:14:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:14:36.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:14:36.411987 3798194 memory.go:184] no items to output this cycle I1122 02:14:36.412147 3798194 cpu.go:275] no items to output this cycle E1122 02:14:46.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:14:46.413255 3798194 memory.go:184] no items to output this cycle I1122 02:14:46.413402 3798194 cpu.go:275] no items to output this cycle E1122 02:14:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:14:56.411956 3798194 memory.go:191] Add success. W1122 02:14:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:14:56.412001 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:14:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:14:56.412350 3798194 cpu.go:282] Add success. I1122 02:14:56.441803 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:14:56.442379 3798194 net.go:642] Add success. I1122 02:14:56.446390 3798194 net.go:778] primary dev: ETH0 I1122 02:14:56.446406 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:14:56.446417 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:14:57.450494 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:14:57.450703 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:14:57.450715 3798194 disk_worker.go:723] disk space is not compliant W1122 02:14:57.450718 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:14:57.458843 3798194 disk_worker.go:509] system disk:vda1 I1122 02:14:57.458883 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:14:58.446016 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:14:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:14:59.450119 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:14:59.450146 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:14:59.471675 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:15:01.338399 3798194 disk_info.go:125] begin check local disk info of client I1122 02:15:01.342376 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:15:01.342385 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004273c0 0xc000427400] E1122 02:15:06.412770 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:15:06.412792 3798194 memory.go:184] no items to output this cycle I1122 02:15:06.412942 3798194 cpu.go:275] no items to output this cycle E1122 02:15:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:15:16.412267 3798194 memory.go:184] no items to output this cycle I1122 02:15:16.412413 3798194 cpu.go:275] no items to output this cycle I1122 02:15:17.590832 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:15:17.590841 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:15:26.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:15:26.412262 3798194 memory.go:191] Add success. I1122 02:15:26.416742 3798194 cpu.go:282] Add success. I1122 02:15:26.441229 3798194 net.go:642] Add success. I1122 02:15:26.442856 3798194 net.go:778] primary dev: ETH0 I1122 02:15:26.442871 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:15:26.442883 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:15:29.450056 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:15:29.450163 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:15:29.450196 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:15:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:15:36.411976 3798194 memory.go:184] no items to output this cycle I1122 02:15:36.412131 3798194 cpu.go:275] no items to output this cycle E1122 02:15:46.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:15:46.412258 3798194 memory.go:184] no items to output this cycle I1122 02:15:46.412404 3798194 cpu.go:275] no items to output this cycle E1122 02:15:56.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:15:56.412272 3798194 memory.go:191] Add success. W1122 02:15:56.412303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:15:56.412316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:15:56.412319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:15:56.416581 3798194 cpu.go:282] Add success. I1122 02:15:56.442265 3798194 event_worker.go:152] Polling the log file for events... I1122 02:15:56.442500 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:15:56.442976 3798194 net.go:642] Add success. I1122 02:15:56.444978 3798194 net.go:778] primary dev: ETH0 I1122 02:15:56.444994 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:15:56.445006 3798194 net.go:659] Add eni metrics success, count: 7 E1122 02:15:57.445310 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 02:15:57.445329 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:15:57.445596 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:15:57.445608 3798194 disk_worker.go:723] disk space is not compliant W1122 02:15:57.445612 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:15:57.448099 3798194 disk_worker.go:509] system disk:vda1 I1122 02:15:57.448150 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 02:15:58.448518 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 02:15:58.448533 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 02:15:59.448983 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 02:15:59.450058 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:15:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:15:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:15:59.471693 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:16:01.346154 3798194 disk_info.go:125] begin check local disk info of client I1122 02:16:01.350061 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:16:01.350070 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003522c0 0xc000352300] E1122 02:16:06.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:16:06.412250 3798194 memory.go:184] no items to output this cycle I1122 02:16:06.412394 3798194 cpu.go:275] no items to output this cycle E1122 02:16:16.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:16:16.412256 3798194 memory.go:184] no items to output this cycle I1122 02:16:16.412401 3798194 cpu.go:275] no items to output this cycle E1122 02:16:26.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:16:26.412282 3798194 memory.go:191] Add success. I1122 02:16:26.417296 3798194 cpu.go:282] Add success. I1122 02:16:26.441763 3798194 net.go:642] Add success. I1122 02:16:26.443355 3798194 net.go:778] primary dev: ETH0 I1122 02:16:26.443369 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:16:26.443383 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:16:29.450036 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:16:29.450127 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:16:29.450157 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:16:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:16:36.412259 3798194 memory.go:184] no items to output this cycle I1122 02:16:36.412608 3798194 cpu.go:275] no items to output this cycle E1122 02:16:46.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:16:46.411989 3798194 memory.go:184] no items to output this cycle I1122 02:16:46.412157 3798194 cpu.go:275] no items to output this cycle E1122 02:16:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:16:56.411947 3798194 memory.go:191] Add success. W1122 02:16:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:16:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:16:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:16:56.412374 3798194 cpu.go:282] Add success. I1122 02:16:56.441798 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:16:56.442388 3798194 net.go:642] Add success. I1122 02:16:56.446676 3798194 net.go:778] primary dev: ETH0 I1122 02:16:56.446690 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:16:56.446703 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:16:57.450768 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:16:57.450930 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:16:57.450985 3798194 disk_worker.go:723] disk space is not compliant W1122 02:16:57.450988 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:16:57.459694 3798194 disk_worker.go:509] system disk:vda1 I1122 02:16:57.459736 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:16:58.446505 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:16:59.450624 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:16:59.450727 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:16:59.450768 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:16:59.471493 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:17:01.353148 3798194 disk_info.go:125] begin check local disk info of client I1122 02:17:01.357354 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:17:01.357364 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ef100 0xc0003ef140] E1122 02:17:06.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:17:06.413261 3798194 memory.go:184] no items to output this cycle I1122 02:17:06.413406 3798194 cpu.go:275] no items to output this cycle E1122 02:17:16.413329 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:17:16.413351 3798194 memory.go:184] no items to output this cycle I1122 02:17:16.413497 3798194 cpu.go:275] no items to output this cycle E1122 02:17:26.411931 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:17:26.411962 3798194 memory.go:191] Add success. I1122 02:17:26.412306 3798194 cpu.go:282] Add success. I1122 02:17:26.441748 3798194 net.go:642] Add success. I1122 02:17:26.445745 3798194 net.go:778] primary dev: ETH0 I1122 02:17:26.445761 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:17:26.445777 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:17:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:17:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:17:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:17:36.412179 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:17:36.412203 3798194 memory.go:184] no items to output this cycle I1122 02:17:36.412347 3798194 cpu.go:275] no items to output this cycle E1122 02:17:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:17:46.411971 3798194 memory.go:184] no items to output this cycle I1122 02:17:46.412130 3798194 cpu.go:275] no items to output this cycle E1122 02:17:56.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:17:56.412261 3798194 memory.go:191] Add success. W1122 02:17:56.412288 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:17:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:17:56.412303 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:17:56.417159 3798194 cpu.go:282] Add success. I1122 02:17:56.441955 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:17:56.442753 3798194 net.go:642] Add success. I1122 02:17:56.444542 3798194 net.go:778] primary dev: ETH0 I1122 02:17:56.444559 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:17:56.444683 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:17:57.450706 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:17:57.450923 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:17:57.450934 3798194 disk_worker.go:723] disk space is not compliant W1122 02:17:57.450937 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:17:57.459312 3798194 disk_worker.go:509] system disk:vda1 I1122 02:17:57.459352 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:17:58.446386 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:17:59.450038 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:17:59.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:17:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:17:59.471645 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:18:01.357470 3798194 disk_info.go:125] begin check local disk info of client I1122 02:18:01.361408 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:18:01.361417 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000070ec0 0xc000070f00] E1122 02:18:06.413254 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:18:06.413278 3798194 memory.go:184] no items to output this cycle I1122 02:18:06.413417 3798194 cpu.go:275] no items to output this cycle E1122 02:18:16.411977 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:18:16.412005 3798194 memory.go:184] no items to output this cycle I1122 02:18:16.412179 3798194 cpu.go:275] no items to output this cycle I1122 02:18:17.592008 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:18:17.592019 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:18:26.411396 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:18:26.411429 3798194 memory.go:191] Add success. I1122 02:18:26.411763 3798194 cpu.go:282] Add success. I1122 02:18:26.441507 3798194 net.go:642] Add success. I1122 02:18:26.445366 3798194 net.go:778] primary dev: ETH0 I1122 02:18:26.445381 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:18:26.445394 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:18:29.449999 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:18:29.450073 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:18:29.450099 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:18:36.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:18:36.411992 3798194 memory.go:184] no items to output this cycle I1122 02:18:36.412153 3798194 cpu.go:275] no items to output this cycle E1122 02:18:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:18:46.411974 3798194 memory.go:184] no items to output this cycle I1122 02:18:46.412131 3798194 cpu.go:275] no items to output this cycle E1122 02:18:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:18:56.411948 3798194 memory.go:191] Add success. W1122 02:18:56.411975 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:18:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:18:56.411989 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:18:56.412342 3798194 cpu.go:282] Add success. I1122 02:18:56.441815 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:18:56.442315 3798194 net.go:642] Add success. I1122 02:18:56.446610 3798194 net.go:778] primary dev: ETH0 I1122 02:18:56.446624 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:18:56.446637 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:18:57.450642 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:18:57.450840 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:18:57.450851 3798194 disk_worker.go:723] disk space is not compliant W1122 02:18:57.450854 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:18:57.458990 3798194 disk_worker.go:509] system disk:vda1 I1122 02:18:57.459033 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:18:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:18:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:18:59.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:18:59.450146 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:18:59.471680 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:19:01.361513 3798194 disk_info.go:125] begin check local disk info of client I1122 02:19:01.365423 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:19:01.365432 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f0880 0xc0003f08c0] E1122 02:19:06.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:19:06.413251 3798194 memory.go:184] no items to output this cycle I1122 02:19:06.413392 3798194 cpu.go:275] no items to output this cycle E1122 02:19:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:19:16.411991 3798194 memory.go:184] no items to output this cycle I1122 02:19:16.412166 3798194 cpu.go:275] no items to output this cycle E1122 02:19:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:19:26.412268 3798194 memory.go:191] Add success. I1122 02:19:26.416651 3798194 cpu.go:282] Add success. I1122 02:19:26.442258 3798194 net.go:642] Add success. I1122 02:19:26.443724 3798194 net.go:778] primary dev: ETH0 I1122 02:19:26.443738 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:19:26.443752 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:19:29.450420 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:19:29.450498 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:19:29.450527 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:19:36.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:19:36.412878 3798194 memory.go:184] no items to output this cycle I1122 02:19:36.413020 3798194 cpu.go:275] no items to output this cycle E1122 02:19:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:19:46.411975 3798194 memory.go:184] no items to output this cycle I1122 02:19:46.412132 3798194 cpu.go:275] no items to output this cycle E1122 02:19:56.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:19:56.412261 3798194 memory.go:191] Add success. W1122 02:19:56.412286 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:19:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:19:56.412301 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:19:56.416522 3798194 cpu.go:282] Add success. I1122 02:19:56.441913 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:19:56.442511 3798194 net.go:642] Add success. I1122 02:19:56.443987 3798194 net.go:778] primary dev: ETH0 I1122 02:19:56.444002 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:19:56.444014 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:19:57.451176 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:19:57.451375 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:19:57.451386 3798194 disk_worker.go:723] disk space is not compliant W1122 02:19:57.451389 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:19:57.459516 3798194 disk_worker.go:509] system disk:vda1 I1122 02:19:57.459556 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:19:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:19:59.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:19:59.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:19:59.450500 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:19:59.471003 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:20:01.369150 3798194 disk_info.go:125] begin check local disk info of client I1122 02:20:01.372992 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:20:01.373001 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000368380 0xc0003683c0] E1122 02:20:06.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:20:06.411983 3798194 memory.go:184] no items to output this cycle I1122 02:20:06.412140 3798194 cpu.go:275] no items to output this cycle E1122 02:20:16.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:20:16.411989 3798194 memory.go:184] no items to output this cycle I1122 02:20:16.412148 3798194 cpu.go:275] no items to output this cycle E1122 02:20:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:20:26.411954 3798194 memory.go:191] Add success. I1122 02:20:26.412305 3798194 cpu.go:282] Add success. I1122 02:20:26.441913 3798194 net.go:642] Add success. I1122 02:20:26.445970 3798194 net.go:778] primary dev: ETH0 I1122 02:20:26.445987 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:20:26.446000 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:20:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:20:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:20:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:20:36.412349 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:20:36.412385 3798194 memory.go:184] no items to output this cycle I1122 02:20:36.412650 3798194 cpu.go:275] no items to output this cycle E1122 02:20:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:20:46.411979 3798194 memory.go:184] no items to output this cycle I1122 02:20:46.412143 3798194 cpu.go:275] no items to output this cycle E1122 02:20:56.412222 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:20:56.412255 3798194 memory.go:191] Add success. W1122 02:20:56.412281 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:20:56.412293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:20:56.412297 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:20:56.417244 3798194 cpu.go:282] Add success. I1122 02:20:56.444552 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:20:56.445024 3798194 net.go:642] Add success. I1122 02:20:56.446594 3798194 net.go:778] primary dev: ETH0 I1122 02:20:56.446611 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:20:56.446623 3798194 net.go:659] Add eni metrics success, count: 7 E1122 02:20:57.450214 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 02:20:57.450229 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:20:57.450416 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:20:57.450425 3798194 disk_worker.go:723] disk space is not compliant W1122 02:20:57.450429 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:20:57.458437 3798194 disk_worker.go:509] system disk:vda1 I1122 02:20:57.458491 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 02:20:58.448184 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 02:20:58.448199 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 02:20:59.448808 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 02:20:59.449882 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:20:59.449937 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:20:59.449956 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:20:59.471515 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:21:01.377144 3798194 disk_info.go:125] begin check local disk info of client I1122 02:21:01.380560 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:21:01.380569 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040c980 0xc00040c9c0] E1122 02:21:06.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:21:06.411987 3798194 memory.go:184] no items to output this cycle I1122 02:21:06.412143 3798194 cpu.go:275] no items to output this cycle E1122 02:21:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:21:16.411978 3798194 memory.go:184] no items to output this cycle I1122 02:21:16.412139 3798194 cpu.go:275] no items to output this cycle I1122 02:21:17.592182 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:21:17.592191 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:21:26.411011 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:21:26.411045 3798194 memory.go:191] Add success. I1122 02:21:26.411402 3798194 cpu.go:282] Add success. I1122 02:21:26.441999 3798194 net.go:642] Add success. I1122 02:21:26.446105 3798194 net.go:778] primary dev: ETH0 I1122 02:21:26.446138 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:21:26.446154 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:21:29.450006 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:21:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:21:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:21:36.412812 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:21:36.412833 3798194 memory.go:184] no items to output this cycle I1122 02:21:36.412981 3798194 cpu.go:275] no items to output this cycle E1122 02:21:46.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:21:46.412878 3798194 memory.go:184] no items to output this cycle I1122 02:21:46.413022 3798194 cpu.go:275] no items to output this cycle E1122 02:21:56.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:21:56.413278 3798194 memory.go:191] Add success. W1122 02:21:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:21:56.413316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:21:56.413319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:21:56.417364 3798194 cpu.go:282] Add success. I1122 02:21:56.441743 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:21:56.442413 3798194 net.go:642] Add success. I1122 02:21:56.444000 3798194 net.go:778] primary dev: ETH0 I1122 02:21:56.444018 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:21:56.444037 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:21:57.450617 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:21:57.450814 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:21:57.450825 3798194 disk_worker.go:723] disk space is not compliant W1122 02:21:57.450828 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:21:57.458875 3798194 disk_worker.go:509] system disk:vda1 I1122 02:21:57.458928 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:21:58.446377 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:21:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:21:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:21:59.450132 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:21:59.471638 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:22:01.385234 3798194 disk_info.go:125] begin check local disk info of client I1122 02:22:01.389165 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:22:01.389174 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071580 0xc0000715c0] E1122 02:22:06.411978 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:22:06.412001 3798194 memory.go:184] no items to output this cycle I1122 02:22:06.412154 3798194 cpu.go:275] no items to output this cycle E1122 02:22:16.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:22:16.413265 3798194 memory.go:184] no items to output this cycle I1122 02:22:16.413411 3798194 cpu.go:275] no items to output this cycle E1122 02:22:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:22:26.411951 3798194 memory.go:191] Add success. I1122 02:22:26.412319 3798194 cpu.go:282] Add success. I1122 02:22:26.442043 3798194 net.go:642] Add success. I1122 02:22:26.446061 3798194 net.go:778] primary dev: ETH0 I1122 02:22:26.446078 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:22:26.446091 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:22:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:22:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:22:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:22:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:22:36.411975 3798194 memory.go:184] no items to output this cycle I1122 02:22:36.412126 3798194 cpu.go:275] no items to output this cycle E1122 02:22:46.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:22:46.413254 3798194 memory.go:184] no items to output this cycle I1122 02:22:46.413399 3798194 cpu.go:275] no items to output this cycle E1122 02:22:56.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:22:56.412273 3798194 memory.go:191] Add success. W1122 02:22:56.412304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:22:56.412315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:22:56.412319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:22:56.416457 3798194 cpu.go:282] Add success. I1122 02:22:56.441934 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:22:56.442523 3798194 net.go:642] Add success. I1122 02:22:56.444232 3798194 net.go:778] primary dev: ETH0 I1122 02:22:56.444248 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:22:56.444260 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:22:57.451330 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:22:57.451495 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:22:57.451551 3798194 disk_worker.go:723] disk space is not compliant W1122 02:22:57.451554 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:22:57.459886 3798194 disk_worker.go:509] system disk:vda1 I1122 02:22:57.459926 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:22:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:22:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:22:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:22:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:22:59.471653 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:23:01.393664 3798194 disk_info.go:125] begin check local disk info of client I1122 02:23:01.397443 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:23:01.397452 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000448040 0xc000448080] E1122 02:23:06.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:23:06.412884 3798194 memory.go:184] no items to output this cycle I1122 02:23:06.413025 3798194 cpu.go:275] no items to output this cycle E1122 02:23:16.412770 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:23:16.412798 3798194 memory.go:184] no items to output this cycle I1122 02:23:16.412942 3798194 cpu.go:275] no items to output this cycle E1122 02:23:26.412261 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:23:26.412306 3798194 memory.go:191] Add success. I1122 02:23:26.416565 3798194 cpu.go:282] Add success. I1122 02:23:26.442281 3798194 net.go:642] Add success. I1122 02:23:26.443836 3798194 net.go:778] primary dev: ETH0 I1122 02:23:26.443851 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:23:26.443865 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:23:29.449636 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:23:29.449723 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:23:29.449756 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:23:36.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:23:36.412252 3798194 memory.go:184] no items to output this cycle I1122 02:23:36.412393 3798194 cpu.go:275] no items to output this cycle E1122 02:23:46.412287 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:23:46.412312 3798194 memory.go:184] no items to output this cycle I1122 02:23:46.412475 3798194 cpu.go:275] no items to output this cycle E1122 02:23:56.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:23:56.411988 3798194 memory.go:191] Add success. W1122 02:23:56.412021 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:23:56.412032 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:23:56.412036 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:23:56.412362 3798194 cpu.go:282] Add success. I1122 02:23:56.441842 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:23:56.442389 3798194 net.go:642] Add success. I1122 02:23:56.446335 3798194 net.go:778] primary dev: ETH0 I1122 02:23:56.446351 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:23:56.446365 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:23:57.451863 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:23:57.452036 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:23:57.452091 3798194 disk_worker.go:723] disk space is not compliant W1122 02:23:57.452094 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:23:57.460925 3798194 disk_worker.go:509] system disk:vda1 I1122 02:23:57.460967 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:23:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:23:59.450487 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:23:59.450567 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:23:59.450599 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:23:59.471101 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:24:01.402094 3798194 disk_info.go:125] begin check local disk info of client I1122 02:24:01.405994 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:24:01.406004 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000388140 0xc000388180] E1122 02:24:06.412765 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:24:06.412786 3798194 memory.go:184] no items to output this cycle I1122 02:24:06.412934 3798194 cpu.go:275] no items to output this cycle E1122 02:24:16.412975 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:24:16.412998 3798194 memory.go:184] no items to output this cycle I1122 02:24:16.413176 3798194 cpu.go:275] no items to output this cycle I1122 02:24:17.592353 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:24:17.592362 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:24:26.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:24:26.412888 3798194 memory.go:191] Add success. I1122 02:24:26.417469 3798194 cpu.go:282] Add success. I1122 02:24:26.442002 3798194 net.go:642] Add success. I1122 02:24:26.443574 3798194 net.go:778] primary dev: ETH0 I1122 02:24:26.443588 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:24:26.443601 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:24:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:24:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:24:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:24:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:24:36.411979 3798194 memory.go:184] no items to output this cycle I1122 02:24:36.412134 3798194 cpu.go:275] no items to output this cycle E1122 02:24:46.412821 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:24:46.412858 3798194 memory.go:184] no items to output this cycle I1122 02:24:46.413004 3798194 cpu.go:275] no items to output this cycle E1122 02:24:56.411935 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:24:56.411967 3798194 memory.go:191] Add success. W1122 02:24:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:24:56.412008 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:24:56.412012 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:24:56.412390 3798194 cpu.go:282] Add success. I1122 02:24:56.441935 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:24:56.442534 3798194 net.go:642] Add success. I1122 02:24:56.446487 3798194 net.go:778] primary dev: ETH0 I1122 02:24:56.446501 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:24:56.446516 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:24:57.450310 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:24:57.450473 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:24:57.450526 3798194 disk_worker.go:723] disk space is not compliant W1122 02:24:57.450529 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:24:57.458645 3798194 disk_worker.go:509] system disk:vda1 I1122 02:24:57.458688 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:24:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:24:59.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:24:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:24:59.450144 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:24:59.471664 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:25:01.410155 3798194 disk_info.go:125] begin check local disk info of client I1122 02:25:01.413991 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:25:01.414000 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fa340 0xc0003fa380] E1122 02:25:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:25:06.411985 3798194 memory.go:184] no items to output this cycle I1122 02:25:06.412146 3798194 cpu.go:275] no items to output this cycle E1122 02:25:16.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:25:16.412884 3798194 memory.go:184] no items to output this cycle I1122 02:25:16.413035 3798194 cpu.go:275] no items to output this cycle E1122 02:25:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:25:26.413268 3798194 memory.go:191] Add success. I1122 02:25:26.417782 3798194 cpu.go:282] Add success. I1122 02:25:26.441266 3798194 net.go:642] Add success. I1122 02:25:26.449040 3798194 net.go:778] primary dev: ETH0 I1122 02:25:26.449056 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:25:26.449073 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:25:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:25:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:25:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:25:36.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:25:36.411984 3798194 memory.go:184] no items to output this cycle I1122 02:25:36.412133 3798194 cpu.go:275] no items to output this cycle E1122 02:25:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:25:46.411980 3798194 memory.go:184] no items to output this cycle I1122 02:25:46.412147 3798194 cpu.go:275] no items to output this cycle E1122 02:25:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:25:56.411946 3798194 memory.go:191] Add success. W1122 02:25:56.411972 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:25:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:25:56.411987 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:25:56.412334 3798194 cpu.go:282] Add success. I1122 02:25:56.441549 3798194 event_worker.go:152] Polling the log file for events... I1122 02:25:56.444366 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:25:56.444952 3798194 net.go:642] Add success. I1122 02:25:56.449219 3798194 net.go:778] primary dev: ETH0 I1122 02:25:56.449234 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:25:56.449246 3798194 net.go:659] Add eni metrics success, count: 7 E1122 02:25:57.450694 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 02:25:57.450708 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:25:57.451102 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:25:57.451136 3798194 disk_worker.go:723] disk space is not compliant W1122 02:25:57.451139 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:25:57.460214 3798194 disk_worker.go:509] system disk:vda1 I1122 02:25:57.460255 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 02:25:58.448090 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 02:25:58.448105 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 02:25:59.448793 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 02:25:59.449868 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:25:59.449919 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:25:59.449940 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:25:59.471466 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:26:01.414263 3798194 disk_info.go:125] begin check local disk info of client I1122 02:26:01.418230 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:26:01.418239 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000471900 0xc000471940] E1122 02:26:06.412763 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:26:06.412785 3798194 memory.go:184] no items to output this cycle I1122 02:26:06.412936 3798194 cpu.go:275] no items to output this cycle E1122 02:26:16.411975 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:26:16.411999 3798194 memory.go:184] no items to output this cycle I1122 02:26:16.412151 3798194 cpu.go:275] no items to output this cycle E1122 02:26:26.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:26:26.412272 3798194 memory.go:191] Add success. I1122 02:26:26.416683 3798194 cpu.go:282] Add success. I1122 02:26:26.442266 3798194 net.go:642] Add success. I1122 02:26:26.443941 3798194 net.go:778] primary dev: ETH0 I1122 02:26:26.443956 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:26:26.443970 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:26:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:26:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:26:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:26:36.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:26:36.411993 3798194 memory.go:184] no items to output this cycle I1122 02:26:36.412150 3798194 cpu.go:275] no items to output this cycle E1122 02:26:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:26:46.411979 3798194 memory.go:184] no items to output this cycle I1122 02:26:46.412141 3798194 cpu.go:275] no items to output this cycle E1122 02:26:56.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:26:56.411959 3798194 memory.go:191] Add success. W1122 02:26:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:26:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:26:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:26:56.412350 3798194 cpu.go:282] Add success. I1122 02:26:56.441815 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:26:56.442402 3798194 net.go:642] Add success. I1122 02:26:56.446502 3798194 net.go:778] primary dev: ETH0 I1122 02:26:56.446519 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:26:56.446532 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:26:57.445284 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:26:57.445492 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:26:57.445556 3798194 disk_worker.go:723] disk space is not compliant W1122 02:26:57.445559 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:26:57.448020 3798194 disk_worker.go:509] system disk:vda1 I1122 02:26:57.448075 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:26:58.445989 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:26:59.450564 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:26:59.450647 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:26:59.450672 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:26:59.471204 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:27:01.418874 3798194 disk_info.go:125] begin check local disk info of client I1122 02:27:01.422752 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:27:01.422766 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003acd00 0xc0003acd40] E1122 02:27:06.413352 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:27:06.413375 3798194 memory.go:184] no items to output this cycle I1122 02:27:06.413530 3798194 cpu.go:275] no items to output this cycle E1122 02:27:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:27:16.413266 3798194 memory.go:184] no items to output this cycle I1122 02:27:16.413411 3798194 cpu.go:275] no items to output this cycle I1122 02:27:17.593434 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:27:17.593443 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:27:26.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:27:26.412274 3798194 memory.go:191] Add success. I1122 02:27:26.416491 3798194 cpu.go:282] Add success. I1122 02:27:26.442025 3798194 net.go:642] Add success. I1122 02:27:26.443684 3798194 net.go:778] primary dev: ETH0 I1122 02:27:26.443697 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:27:26.443710 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:27:29.450407 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:27:29.450489 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:27:29.450519 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:27:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:27:36.413260 3798194 memory.go:184] no items to output this cycle I1122 02:27:36.413408 3798194 cpu.go:275] no items to output this cycle E1122 02:27:46.412939 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:27:46.412960 3798194 memory.go:184] no items to output this cycle I1122 02:27:46.413136 3798194 cpu.go:275] no items to output this cycle E1122 02:27:56.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:27:56.412261 3798194 memory.go:191] Add success. W1122 02:27:56.412293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:27:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:27:56.412308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:27:56.416903 3798194 cpu.go:282] Add success. I1122 02:27:56.441332 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:27:56.441906 3798194 net.go:642] Add success. I1122 02:27:56.443501 3798194 net.go:778] primary dev: ETH0 I1122 02:27:56.443517 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:27:56.443538 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:27:57.451025 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:27:57.451208 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:27:57.451263 3798194 disk_worker.go:723] disk space is not compliant W1122 02:27:57.451266 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:27:57.459722 3798194 disk_worker.go:509] system disk:vda1 I1122 02:27:57.459762 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:27:58.446389 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:27:59.450403 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:27:59.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:27:59.450508 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:27:59.471026 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:28:01.423274 3798194 disk_info.go:125] begin check local disk info of client I1122 02:28:01.427177 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:28:01.427187 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000323400 0xc000323440] E1122 02:28:06.411982 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:28:06.412008 3798194 memory.go:184] no items to output this cycle I1122 02:28:06.412160 3798194 cpu.go:275] no items to output this cycle E1122 02:28:16.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:28:16.412246 3798194 memory.go:184] no items to output this cycle I1122 02:28:16.412403 3798194 cpu.go:275] no items to output this cycle E1122 02:28:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:28:26.411946 3798194 memory.go:191] Add success. I1122 02:28:26.412300 3798194 cpu.go:282] Add success. I1122 02:28:26.441759 3798194 net.go:642] Add success. I1122 02:28:26.445803 3798194 net.go:778] primary dev: ETH0 I1122 02:28:26.445820 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:28:26.445832 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:28:29.450050 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:28:29.450132 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:28:29.450162 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:28:36.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:28:36.413269 3798194 memory.go:184] no items to output this cycle I1122 02:28:36.413418 3798194 cpu.go:275] no items to output this cycle E1122 02:28:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:28:46.411980 3798194 memory.go:184] no items to output this cycle I1122 02:28:46.412139 3798194 cpu.go:275] no items to output this cycle E1122 02:28:56.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:28:56.412279 3798194 memory.go:191] Add success. W1122 02:28:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:28:56.412323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:28:56.412327 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:28:56.417262 3798194 cpu.go:282] Add success. I1122 02:28:56.440956 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:28:56.441548 3798194 net.go:642] Add success. I1122 02:28:56.443274 3798194 net.go:778] primary dev: ETH0 I1122 02:28:56.443288 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:28:56.443301 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:28:57.450426 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:28:57.450633 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:28:57.450644 3798194 disk_worker.go:723] disk space is not compliant W1122 02:28:57.450647 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:28:57.458717 3798194 disk_worker.go:509] system disk:vda1 I1122 02:28:57.458761 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:28:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:28:59.449882 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:28:59.449969 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:28:59.449995 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:28:59.471536 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:29:01.427244 3798194 disk_info.go:125] begin check local disk info of client I1122 02:29:01.430812 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:29:01.430822 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ec00 0xc00030ec40] E1122 02:29:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:29:06.411977 3798194 memory.go:184] no items to output this cycle I1122 02:29:06.412127 3798194 cpu.go:275] no items to output this cycle E1122 02:29:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:29:16.411976 3798194 memory.go:184] no items to output this cycle I1122 02:29:16.412132 3798194 cpu.go:275] no items to output this cycle E1122 02:29:26.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:29:26.413275 3798194 memory.go:191] Add success. I1122 02:29:26.417337 3798194 cpu.go:282] Add success. I1122 02:29:26.441747 3798194 net.go:642] Add success. I1122 02:29:26.443366 3798194 net.go:778] primary dev: ETH0 I1122 02:29:26.443380 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:29:26.443392 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:29:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:29:29.450122 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:29:29.450152 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:29:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:29:36.411972 3798194 memory.go:184] no items to output this cycle I1122 02:29:36.412123 3798194 cpu.go:275] no items to output this cycle E1122 02:29:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:29:46.411975 3798194 memory.go:184] no items to output this cycle I1122 02:29:46.412138 3798194 cpu.go:275] no items to output this cycle E1122 02:29:56.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:29:56.412883 3798194 memory.go:191] Add success. W1122 02:29:56.412909 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:29:56.412921 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:29:56.412924 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:29:56.417216 3798194 cpu.go:282] Add success. I1122 02:29:56.441613 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:29:56.442199 3798194 net.go:642] Add success. I1122 02:29:56.443663 3798194 net.go:778] primary dev: ETH0 I1122 02:29:56.443677 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:29:56.443696 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:29:57.450524 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:29:57.450684 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:29:57.450736 3798194 disk_worker.go:723] disk space is not compliant W1122 02:29:57.450739 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:29:57.459133 3798194 disk_worker.go:509] system disk:vda1 I1122 02:29:57.459175 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:29:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:29:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:29:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:29:59.450132 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:29:59.471700 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:30:01.430888 3798194 disk_info.go:125] begin check local disk info of client I1122 02:30:01.434597 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:30:01.434608 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ffc40 0xc0003ffc80] E1122 02:30:06.419252 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:30:06.419277 3798194 memory.go:184] no items to output this cycle I1122 02:30:06.419498 3798194 cpu.go:275] no items to output this cycle E1122 02:30:16.416257 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:30:16.416279 3798194 memory.go:184] no items to output this cycle I1122 02:30:16.416426 3798194 cpu.go:275] no items to output this cycle I1122 02:30:17.593597 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:30:17.593605 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:30:26.411380 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:30:26.411412 3798194 memory.go:191] Add success. I1122 02:30:26.411779 3798194 cpu.go:282] Add success. I1122 02:30:26.441286 3798194 net.go:642] Add success. I1122 02:30:26.445548 3798194 net.go:778] primary dev: ETH0 I1122 02:30:26.445563 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:30:26.445577 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:30:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:30:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:30:29.450512 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:30:36.412429 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:30:36.412451 3798194 memory.go:184] no items to output this cycle I1122 02:30:36.412596 3798194 cpu.go:275] no items to output this cycle E1122 02:30:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:30:46.411974 3798194 memory.go:184] no items to output this cycle I1122 02:30:46.412134 3798194 cpu.go:275] no items to output this cycle E1122 02:30:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:30:56.411957 3798194 memory.go:191] Add success. W1122 02:30:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:30:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:30:56.412006 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:30:56.412353 3798194 cpu.go:282] Add success. I1122 02:30:56.444404 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:30:56.444982 3798194 net.go:642] Add success. I1122 02:30:56.448704 3798194 net.go:778] primary dev: ETH0 I1122 02:30:56.448718 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:30:56.448732 3798194 net.go:659] Add eni metrics success, count: 7 E1122 02:30:57.450407 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 02:30:57.450421 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:30:57.450628 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:30:57.450637 3798194 disk_worker.go:723] disk space is not compliant W1122 02:30:57.450640 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:30:57.458725 3798194 disk_worker.go:509] system disk:vda1 I1122 02:30:57.458768 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 02:30:58.447843 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 02:30:58.447858 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 02:30:59.448612 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 02:30:59.449689 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:30:59.449741 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:30:59.449760 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:30:59.471302 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:31:01.434966 3798194 disk_info.go:125] begin check local disk info of client I1122 02:31:01.438526 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:31:01.438534 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000373100 0xc000373140] E1122 02:31:06.412762 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:31:06.412784 3798194 memory.go:184] no items to output this cycle I1122 02:31:06.412934 3798194 cpu.go:275] no items to output this cycle E1122 02:31:16.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:31:16.412267 3798194 memory.go:184] no items to output this cycle I1122 02:31:16.412437 3798194 cpu.go:275] no items to output this cycle E1122 02:31:26.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:31:26.412267 3798194 memory.go:191] Add success. I1122 02:31:26.417230 3798194 cpu.go:282] Add success. I1122 02:31:26.441738 3798194 net.go:642] Add success. I1122 02:31:26.443178 3798194 net.go:778] primary dev: ETH0 I1122 02:31:26.443193 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:31:26.443206 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:31:29.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:31:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:31:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:31:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:31:36.412259 3798194 memory.go:184] no items to output this cycle I1122 02:31:36.412407 3798194 cpu.go:275] no items to output this cycle E1122 02:31:46.412865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:31:46.412895 3798194 memory.go:184] no items to output this cycle I1122 02:31:46.413038 3798194 cpu.go:275] no items to output this cycle E1122 02:31:56.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:31:56.412272 3798194 memory.go:191] Add success. W1122 02:31:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:31:56.412316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:31:56.412320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:31:56.416538 3798194 cpu.go:282] Add success. I1122 02:31:56.440927 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:31:56.441543 3798194 net.go:642] Add success. I1122 02:31:56.443188 3798194 net.go:778] primary dev: ETH0 I1122 02:31:56.443205 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:31:56.443219 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:31:57.445241 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:31:57.445467 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:31:57.445543 3798194 disk_worker.go:723] disk space is not compliant W1122 02:31:57.445547 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:31:57.448034 3798194 disk_worker.go:509] system disk:vda1 I1122 02:31:57.448076 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:31:58.446385 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:31:59.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:31:59.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:31:59.450505 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:31:59.471002 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:32:01.439258 3798194 disk_info.go:125] begin check local disk info of client I1122 02:32:01.443070 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:32:01.443079 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ed300 0xc0003ed340] E1122 02:32:06.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:32:06.413255 3798194 memory.go:184] no items to output this cycle I1122 02:32:06.413415 3798194 cpu.go:275] no items to output this cycle E1122 02:32:16.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:32:16.411986 3798194 memory.go:184] no items to output this cycle I1122 02:32:16.412144 3798194 cpu.go:275] no items to output this cycle E1122 02:32:26.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:32:26.411960 3798194 memory.go:191] Add success. I1122 02:32:26.412311 3798194 cpu.go:282] Add success. I1122 02:32:26.441750 3798194 net.go:642] Add success. I1122 02:32:26.445673 3798194 net.go:778] primary dev: ETH0 I1122 02:32:26.445688 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:32:26.445703 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:32:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:32:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:32:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:32:36.412130 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:32:36.412153 3798194 memory.go:184] no items to output this cycle I1122 02:32:36.412295 3798194 cpu.go:275] no items to output this cycle E1122 02:32:46.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:32:46.412254 3798194 memory.go:184] no items to output this cycle I1122 02:32:46.412399 3798194 cpu.go:275] no items to output this cycle E1122 02:32:56.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:32:56.411958 3798194 memory.go:191] Add success. W1122 02:32:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:32:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:32:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:32:56.412359 3798194 cpu.go:282] Add success. I1122 02:32:56.441815 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:32:56.442419 3798194 net.go:642] Add success. I1122 02:32:56.446527 3798194 net.go:778] primary dev: ETH0 I1122 02:32:56.446541 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:32:56.446557 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:32:57.450442 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:32:57.450599 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:32:57.450651 3798194 disk_worker.go:723] disk space is not compliant W1122 02:32:57.450654 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:32:57.458823 3798194 disk_worker.go:509] system disk:vda1 I1122 02:32:57.458879 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:32:58.446010 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:32:59.453909 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:32:59.453986 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:32:59.454012 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:32:59.471526 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:33:01.444002 3798194 disk_info.go:125] begin check local disk info of client I1122 02:33:01.447547 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:33:01.447556 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00049f440 0xc00049f480] E1122 02:33:06.412766 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:33:06.412787 3798194 memory.go:184] no items to output this cycle I1122 02:33:06.412936 3798194 cpu.go:275] no items to output this cycle E1122 02:33:16.412340 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:33:16.412361 3798194 memory.go:184] no items to output this cycle I1122 02:33:16.412505 3798194 cpu.go:275] no items to output this cycle I1122 02:33:17.593766 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:33:17.593775 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:33:26.412034 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:33:26.412065 3798194 memory.go:191] Add success. I1122 02:33:26.412427 3798194 cpu.go:282] Add success. I1122 02:33:26.442154 3798194 net.go:642] Add success. I1122 02:33:26.446159 3798194 net.go:778] primary dev: ETH0 I1122 02:33:26.446174 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:33:26.446190 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:33:29.450385 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:33:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:33:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:33:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:33:36.411977 3798194 memory.go:184] no items to output this cycle I1122 02:33:36.412131 3798194 cpu.go:275] no items to output this cycle E1122 02:33:46.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:33:46.412252 3798194 memory.go:184] no items to output this cycle I1122 02:33:46.412399 3798194 cpu.go:275] no items to output this cycle E1122 02:33:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:33:56.413268 3798194 memory.go:191] Add success. W1122 02:33:56.413294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:33:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:33:56.413309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:33:56.417847 3798194 cpu.go:282] Add success. I1122 02:33:56.441267 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:33:56.441827 3798194 net.go:642] Add success. I1122 02:33:56.443380 3798194 net.go:778] primary dev: ETH0 I1122 02:33:56.443395 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:33:56.443408 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:33:57.451039 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:33:57.451227 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:33:57.451280 3798194 disk_worker.go:723] disk space is not compliant W1122 02:33:57.451283 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:33:57.459726 3798194 disk_worker.go:509] system disk:vda1 I1122 02:33:57.459769 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:33:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:33:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:33:59.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:33:59.450497 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:33:59.471005 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:34:01.448008 3798194 disk_info.go:125] begin check local disk info of client I1122 02:34:01.451593 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:34:01.451601 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003da200 0xc0003da240] E1122 02:34:06.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:34:06.411970 3798194 memory.go:184] no items to output this cycle I1122 02:34:06.412124 3798194 cpu.go:275] no items to output this cycle E1122 02:34:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:34:16.411976 3798194 memory.go:184] no items to output this cycle I1122 02:34:16.412144 3798194 cpu.go:275] no items to output this cycle E1122 02:34:26.412873 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:34:26.412912 3798194 memory.go:191] Add success. I1122 02:34:26.417370 3798194 cpu.go:282] Add success. I1122 02:34:26.441926 3798194 net.go:642] Add success. I1122 02:34:26.443435 3798194 net.go:778] primary dev: ETH0 I1122 02:34:26.443450 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:34:26.443572 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:34:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:34:29.450130 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:34:29.450162 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:34:36.413187 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:34:36.413209 3798194 memory.go:184] no items to output this cycle I1122 02:34:36.413354 3798194 cpu.go:275] no items to output this cycle E1122 02:34:46.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:34:46.412265 3798194 memory.go:184] no items to output this cycle I1122 02:34:46.412410 3798194 cpu.go:275] no items to output this cycle E1122 02:34:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:34:56.411949 3798194 memory.go:191] Add success. W1122 02:34:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:34:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:34:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:34:56.412342 3798194 cpu.go:282] Add success. I1122 02:34:56.441803 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:34:56.442303 3798194 net.go:642] Add success. I1122 02:34:56.446926 3798194 net.go:778] primary dev: ETH0 I1122 02:34:56.446941 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:34:56.446954 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:34:57.450816 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:34:57.451038 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:34:57.451048 3798194 disk_worker.go:723] disk space is not compliant W1122 02:34:57.451051 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:34:57.460030 3798194 disk_worker.go:509] system disk:vda1 I1122 02:34:57.460073 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:34:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:34:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:34:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:34:59.450133 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:34:59.471649 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:35:01.452420 3798194 disk_info.go:125] begin check local disk info of client I1122 02:35:01.456061 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:35:01.456070 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00049c500 0xc00049c540] E1122 02:35:06.412759 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:35:06.412783 3798194 memory.go:184] no items to output this cycle I1122 02:35:06.412933 3798194 cpu.go:275] no items to output this cycle E1122 02:35:16.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:35:16.412247 3798194 memory.go:184] no items to output this cycle I1122 02:35:16.412391 3798194 cpu.go:275] no items to output this cycle E1122 02:35:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:35:26.411957 3798194 memory.go:191] Add success. I1122 02:35:26.412304 3798194 cpu.go:282] Add success. I1122 02:35:26.442000 3798194 net.go:642] Add success. I1122 02:35:26.446252 3798194 net.go:778] primary dev: ETH0 I1122 02:35:26.446267 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:35:26.446282 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:35:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:35:29.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:35:29.450150 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:35:36.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:35:36.412268 3798194 memory.go:184] no items to output this cycle I1122 02:35:36.412416 3798194 cpu.go:275] no items to output this cycle E1122 02:35:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:35:46.411984 3798194 memory.go:184] no items to output this cycle I1122 02:35:46.412147 3798194 cpu.go:275] no items to output this cycle E1122 02:35:56.411909 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:35:56.411940 3798194 memory.go:191] Add success. W1122 02:35:56.411967 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:35:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:35:56.411982 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:35:56.412322 3798194 cpu.go:282] Add success. I1122 02:35:56.441542 3798194 event_worker.go:152] Polling the log file for events... I1122 02:35:56.444626 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:35:56.445099 3798194 net.go:642] Add success. I1122 02:35:56.449082 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795284356, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 02:35:56.449236 3798194 net.go:778] primary dev: ETH0 I1122 02:35:56.449250 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:35:56.449262 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:35:56.465635 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 02:35:57.445387 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 02:35:57.445406 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:35:57.445647 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:35:57.445657 3798194 disk_worker.go:723] disk space is not compliant W1122 02:35:57.445660 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:35:57.448190 3798194 disk_worker.go:509] system disk:vda1 I1122 02:35:57.448228 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 02:35:58.447792 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 02:35:58.447808 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 02:35:59.448616 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 02:35:59.449661 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:35:59.449712 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:35:59.449736 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:35:59.471190 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:36:01.457056 3798194 disk_info.go:125] begin check local disk info of client I1122 02:36:01.460924 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:36:01.460933 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004be5c0 0xc0004be600] E1122 02:36:06.412762 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:36:06.412784 3798194 memory.go:184] no items to output this cycle I1122 02:36:06.412932 3798194 cpu.go:275] no items to output this cycle E1122 02:36:16.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:36:16.411987 3798194 memory.go:184] no items to output this cycle I1122 02:36:16.412149 3798194 cpu.go:275] no items to output this cycle I1122 02:36:17.605221 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:36:17.605230 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:36:26.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:36:26.412896 3798194 memory.go:191] Add success. I1122 02:36:26.417082 3798194 cpu.go:282] Add success. I1122 02:36:26.441627 3798194 net.go:642] Add success. I1122 02:36:26.443238 3798194 net.go:778] primary dev: ETH0 I1122 02:36:26.443253 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:36:26.443275 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:36:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:36:29.450127 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:36:29.450154 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:36:36.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:36:36.411987 3798194 memory.go:184] no items to output this cycle I1122 02:36:36.412147 3798194 cpu.go:275] no items to output this cycle E1122 02:36:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:36:46.411981 3798194 memory.go:184] no items to output this cycle I1122 02:36:46.412158 3798194 cpu.go:275] no items to output this cycle E1122 02:36:56.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:36:56.412892 3798194 memory.go:191] Add success. W1122 02:36:56.412924 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:36:56.412935 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:36:56.412939 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:36:56.417814 3798194 cpu.go:282] Add success. I1122 02:36:56.441197 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:36:56.441678 3798194 net.go:642] Add success. I1122 02:36:56.443252 3798194 net.go:778] primary dev: ETH0 I1122 02:36:56.443267 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:36:56.443278 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:36:57.450700 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:36:57.450865 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:36:57.450924 3798194 disk_worker.go:723] disk space is not compliant W1122 02:36:57.450927 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:36:57.459177 3798194 disk_worker.go:509] system disk:vda1 I1122 02:36:57.459221 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:36:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:36:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:36:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:36:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:36:59.471651 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:37:01.461023 3798194 disk_info.go:125] begin check local disk info of client I1122 02:37:01.464853 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:37:01.464863 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040d780 0xc00040d7c0] E1122 02:37:06.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:37:06.413258 3798194 memory.go:184] no items to output this cycle I1122 02:37:06.413403 3798194 cpu.go:275] no items to output this cycle E1122 02:37:16.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:37:16.413255 3798194 memory.go:184] no items to output this cycle I1122 02:37:16.413408 3798194 cpu.go:275] no items to output this cycle E1122 02:37:26.411935 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:37:26.411973 3798194 memory.go:191] Add success. I1122 02:37:26.412347 3798194 cpu.go:282] Add success. I1122 02:37:26.441993 3798194 net.go:642] Add success. I1122 02:37:26.446179 3798194 net.go:778] primary dev: ETH0 I1122 02:37:26.446195 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:37:26.446208 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:37:29.450030 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:37:29.450130 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:37:29.450168 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:37:36.412770 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:37:36.412790 3798194 memory.go:184] no items to output this cycle I1122 02:37:36.412964 3798194 cpu.go:275] no items to output this cycle E1122 02:37:46.411973 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:37:46.411997 3798194 memory.go:184] no items to output this cycle I1122 02:37:46.412179 3798194 cpu.go:275] no items to output this cycle E1122 02:37:56.411944 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:37:56.411978 3798194 memory.go:191] Add success. W1122 02:37:56.412013 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:37:56.412024 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:37:56.412027 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:37:56.412395 3798194 cpu.go:282] Add success. I1122 02:37:56.441933 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:37:56.442548 3798194 net.go:642] Add success. I1122 02:37:56.446480 3798194 net.go:778] primary dev: ETH0 I1122 02:37:56.446494 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:37:56.446507 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:37:57.451039 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:37:57.451284 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:37:57.451295 3798194 disk_worker.go:723] disk space is not compliant W1122 02:37:57.451298 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:37:57.459840 3798194 disk_worker.go:509] system disk:vda1 I1122 02:37:57.459897 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:37:58.446543 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:37:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:37:59.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:37:59.450507 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:37:59.471057 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:38:01.464959 3798194 disk_info.go:125] begin check local disk info of client I1122 02:38:01.468500 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:38:01.468508 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00015d480 0xc00015d4c0] E1122 02:38:06.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:38:06.411979 3798194 memory.go:184] no items to output this cycle I1122 02:38:06.412131 3798194 cpu.go:275] no items to output this cycle E1122 02:38:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:38:16.413263 3798194 memory.go:184] no items to output this cycle I1122 02:38:16.413413 3798194 cpu.go:275] no items to output this cycle E1122 02:38:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:38:26.411952 3798194 memory.go:191] Add success. I1122 02:38:26.412315 3798194 cpu.go:282] Add success. I1122 02:38:26.441921 3798194 net.go:642] Add success. I1122 02:38:26.445808 3798194 net.go:778] primary dev: ETH0 I1122 02:38:26.445822 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:38:26.445835 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:38:29.450029 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:38:29.450131 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:38:29.450157 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:38:36.412765 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:38:36.412788 3798194 memory.go:184] no items to output this cycle I1122 02:38:36.412939 3798194 cpu.go:275] no items to output this cycle E1122 02:38:46.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:38:46.411989 3798194 memory.go:184] no items to output this cycle I1122 02:38:46.412146 3798194 cpu.go:275] no items to output this cycle E1122 02:38:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:38:56.411953 3798194 memory.go:191] Add success. W1122 02:38:56.412098 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:38:56.412125 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:38:56.412132 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:38:56.412566 3798194 cpu.go:282] Add success. I1122 02:38:56.440970 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:38:56.441567 3798194 net.go:642] Add success. I1122 02:38:56.445684 3798194 net.go:778] primary dev: ETH0 I1122 02:38:56.445700 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:38:56.445713 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:38:57.450751 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:38:57.450952 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:38:57.450963 3798194 disk_worker.go:723] disk space is not compliant W1122 02:38:57.450966 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:38:57.459216 3798194 disk_worker.go:509] system disk:vda1 I1122 02:38:57.459257 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:38:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:38:59.450102 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:38:59.450209 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:38:59.450239 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:38:59.470711 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:39:01.469135 3798194 disk_info.go:125] begin check local disk info of client I1122 02:39:01.472779 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:39:01.472787 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ed00 0xc00030ed40] E1122 02:39:06.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:39:06.411990 3798194 memory.go:184] no items to output this cycle I1122 02:39:06.412144 3798194 cpu.go:275] no items to output this cycle E1122 02:39:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:39:16.412259 3798194 memory.go:184] no items to output this cycle I1122 02:39:16.412410 3798194 cpu.go:275] no items to output this cycle I1122 02:39:17.606231 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:39:17.606239 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:39:26.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:39:26.412269 3798194 memory.go:191] Add success. I1122 02:39:26.416602 3798194 cpu.go:282] Add success. I1122 02:39:26.442157 3798194 net.go:642] Add success. I1122 02:39:26.443670 3798194 net.go:778] primary dev: ETH0 I1122 02:39:26.443684 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:39:26.443697 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:39:29.450407 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:39:29.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:39:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:39:36.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:39:36.411988 3798194 memory.go:184] no items to output this cycle I1122 02:39:36.412142 3798194 cpu.go:275] no items to output this cycle E1122 02:39:46.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:39:46.413256 3798194 memory.go:184] no items to output this cycle I1122 02:39:46.413409 3798194 cpu.go:275] no items to output this cycle E1122 02:39:56.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:39:56.412897 3798194 memory.go:191] Add success. W1122 02:39:56.412927 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:39:56.412938 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:39:56.412942 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:39:56.417497 3798194 cpu.go:282] Add success. I1122 02:39:56.441888 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:39:56.442392 3798194 net.go:642] Add success. I1122 02:39:56.443975 3798194 net.go:778] primary dev: ETH0 I1122 02:39:56.443988 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:39:56.444001 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:39:57.451475 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:39:57.451683 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:39:57.451693 3798194 disk_worker.go:723] disk space is not compliant W1122 02:39:57.451696 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:39:57.460907 3798194 disk_worker.go:509] system disk:vda1 I1122 02:39:57.460946 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:39:58.446372 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:39:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:39:59.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:39:59.450512 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:39:59.471025 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:40:01.472889 3798194 disk_info.go:125] begin check local disk info of client I1122 02:40:01.476834 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:40:01.476844 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ef680 0xc0003ef6c0] E1122 02:40:06.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:40:06.411974 3798194 memory.go:184] no items to output this cycle I1122 02:40:06.412130 3798194 cpu.go:275] no items to output this cycle E1122 02:40:16.416286 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:40:16.416315 3798194 memory.go:184] no items to output this cycle I1122 02:40:16.416533 3798194 cpu.go:275] no items to output this cycle E1122 02:40:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:40:26.411953 3798194 memory.go:191] Add success. I1122 02:40:26.412315 3798194 cpu.go:282] Add success. I1122 02:40:26.441909 3798194 net.go:642] Add success. I1122 02:40:26.446234 3798194 net.go:778] primary dev: ETH0 I1122 02:40:26.446248 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:40:26.446261 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:40:29.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:40:29.450121 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:40:29.450148 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:40:36.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:40:36.412261 3798194 memory.go:184] no items to output this cycle I1122 02:40:36.412416 3798194 cpu.go:275] no items to output this cycle E1122 02:40:46.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:40:46.411973 3798194 memory.go:184] no items to output this cycle I1122 02:40:46.412135 3798194 cpu.go:275] no items to output this cycle E1122 02:40:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:40:56.411948 3798194 memory.go:191] Add success. W1122 02:40:56.411981 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:40:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:40:56.411996 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:40:56.412359 3798194 cpu.go:282] Add success. I1122 02:40:56.444741 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:40:56.445257 3798194 net.go:642] Add success. I1122 02:40:56.449297 3798194 net.go:778] primary dev: ETH0 I1122 02:40:56.449311 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:40:56.449323 3798194 net.go:659] Add eni metrics success, count: 7 E1122 02:40:57.451021 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 02:40:57.451037 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:40:57.451267 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:40:57.451277 3798194 disk_worker.go:723] disk space is not compliant W1122 02:40:57.451280 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:40:57.460074 3798194 disk_worker.go:509] system disk:vda1 I1122 02:40:57.460134 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 02:40:58.447814 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 02:40:58.447829 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 02:40:59.448520 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 02:40:59.450647 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:40:59.450698 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:40:59.450718 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:40:59.471242 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:41:01.476945 3798194 disk_info.go:125] begin check local disk info of client I1122 02:41:01.480493 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:41:01.480502 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037e480 0xc00037e4c0] E1122 02:41:06.412090 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:41:06.412127 3798194 memory.go:184] no items to output this cycle I1122 02:41:06.412270 3798194 cpu.go:275] no items to output this cycle E1122 02:41:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:41:16.411985 3798194 memory.go:184] no items to output this cycle I1122 02:41:16.412161 3798194 cpu.go:275] no items to output this cycle E1122 02:41:26.411929 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:41:26.411965 3798194 memory.go:191] Add success. I1122 02:41:26.412335 3798194 cpu.go:282] Add success. I1122 02:41:26.441978 3798194 net.go:642] Add success. I1122 02:41:26.446009 3798194 net.go:778] primary dev: ETH0 I1122 02:41:26.446026 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:41:26.446039 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:41:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:41:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:41:29.450145 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:41:36.413222 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:41:36.413244 3798194 memory.go:184] no items to output this cycle I1122 02:41:36.413390 3798194 cpu.go:275] no items to output this cycle E1122 02:41:46.411933 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:41:46.411954 3798194 memory.go:184] no items to output this cycle I1122 02:41:46.412097 3798194 cpu.go:275] no items to output this cycle E1122 02:41:56.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:41:56.412884 3798194 memory.go:191] Add success. W1122 02:41:56.412918 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:41:56.412930 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:41:56.412935 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:41:56.417108 3798194 cpu.go:282] Add success. I1122 02:41:56.441422 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:41:56.441992 3798194 net.go:642] Add success. I1122 02:41:56.444101 3798194 net.go:778] primary dev: ETH0 I1122 02:41:56.444128 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:41:56.444140 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:41:57.450796 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:41:57.450963 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:41:57.451022 3798194 disk_worker.go:723] disk space is not compliant W1122 02:41:57.451026 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:41:57.459873 3798194 disk_worker.go:509] system disk:vda1 I1122 02:41:57.459912 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:41:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:41:59.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:41:59.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:41:59.450509 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:41:59.471037 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:42:01.481560 3798194 disk_info.go:125] begin check local disk info of client I1122 02:42:01.485143 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:42:01.485152 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002f5440 0xc0002f5480] E1122 02:42:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:42:06.411979 3798194 memory.go:184] no items to output this cycle I1122 02:42:06.412147 3798194 cpu.go:275] no items to output this cycle E1122 02:42:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:42:16.411982 3798194 memory.go:184] no items to output this cycle I1122 02:42:16.412140 3798194 cpu.go:275] no items to output this cycle I1122 02:42:17.610251 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:42:17.610263 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:42:26.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:42:26.412270 3798194 memory.go:191] Add success. I1122 02:42:26.416288 3798194 cpu.go:282] Add success. I1122 02:42:26.441741 3798194 net.go:642] Add success. I1122 02:42:26.443231 3798194 net.go:778] primary dev: ETH0 I1122 02:42:26.443245 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:42:26.443259 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:42:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:42:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:42:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:42:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:42:36.411977 3798194 memory.go:184] no items to output this cycle I1122 02:42:36.412142 3798194 cpu.go:275] no items to output this cycle E1122 02:42:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:42:46.411987 3798194 memory.go:184] no items to output this cycle I1122 02:42:46.412148 3798194 cpu.go:275] no items to output this cycle E1122 02:42:56.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:42:56.411960 3798194 memory.go:191] Add success. W1122 02:42:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:42:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:42:56.412007 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:42:56.412445 3798194 cpu.go:282] Add success. I1122 02:42:56.441927 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:42:56.442533 3798194 net.go:642] Add success. I1122 02:42:56.446643 3798194 net.go:778] primary dev: ETH0 I1122 02:42:56.446658 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:42:56.446672 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:42:57.450441 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:42:57.450650 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:42:57.450661 3798194 disk_worker.go:723] disk space is not compliant W1122 02:42:57.450664 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:42:57.458790 3798194 disk_worker.go:509] system disk:vda1 I1122 02:42:57.458834 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:42:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:42:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:42:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:42:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:42:59.471660 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:43:01.486211 3798194 disk_info.go:125] begin check local disk info of client I1122 02:43:01.489754 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:43:01.489763 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037edc0 0xc00037ee00] E1122 02:43:06.412779 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:43:06.412802 3798194 memory.go:184] no items to output this cycle I1122 02:43:06.412955 3798194 cpu.go:275] no items to output this cycle E1122 02:43:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:43:16.412261 3798194 memory.go:184] no items to output this cycle I1122 02:43:16.412411 3798194 cpu.go:275] no items to output this cycle E1122 02:43:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:43:26.411955 3798194 memory.go:191] Add success. I1122 02:43:26.412309 3798194 cpu.go:282] Add success. I1122 02:43:26.441750 3798194 net.go:642] Add success. I1122 02:43:26.445648 3798194 net.go:778] primary dev: ETH0 I1122 02:43:26.445662 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:43:26.445676 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:43:29.450008 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:43:29.450085 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:43:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:43:36.413162 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:43:36.413184 3798194 memory.go:184] no items to output this cycle I1122 02:43:36.413325 3798194 cpu.go:275] no items to output this cycle E1122 02:43:46.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:43:46.412253 3798194 memory.go:184] no items to output this cycle I1122 02:43:46.412402 3798194 cpu.go:275] no items to output this cycle E1122 02:43:56.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:43:56.412266 3798194 memory.go:191] Add success. W1122 02:43:56.412298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:43:56.412310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:43:56.412313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:43:56.416676 3798194 cpu.go:282] Add success. I1122 02:43:56.440963 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:43:56.441573 3798194 net.go:642] Add success. I1122 02:43:56.443027 3798194 net.go:778] primary dev: ETH0 I1122 02:43:56.443042 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:43:56.443056 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:43:57.451516 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:43:57.451682 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:43:57.451739 3798194 disk_worker.go:723] disk space is not compliant W1122 02:43:57.451743 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:43:57.460213 3798194 disk_worker.go:509] system disk:vda1 I1122 02:43:57.460255 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:43:58.446394 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:43:59.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:43:59.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:43:59.450506 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:43:59.471027 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:44:01.490611 3798194 disk_info.go:125] begin check local disk info of client I1122 02:44:01.494128 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:44:01.494138 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00043de00 0xc00043de40] E1122 02:44:06.413276 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:44:06.413307 3798194 memory.go:184] no items to output this cycle I1122 02:44:06.413509 3798194 cpu.go:275] no items to output this cycle E1122 02:44:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:44:16.411978 3798194 memory.go:184] no items to output this cycle I1122 02:44:16.412131 3798194 cpu.go:275] no items to output this cycle E1122 02:44:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:44:26.411957 3798194 memory.go:191] Add success. I1122 02:44:26.412316 3798194 cpu.go:282] Add success. I1122 02:44:26.441942 3798194 net.go:642] Add success. I1122 02:44:26.446065 3798194 net.go:778] primary dev: ETH0 I1122 02:44:26.446080 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:44:26.446093 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:44:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:44:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:44:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:44:36.412083 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:44:36.412127 3798194 memory.go:184] no items to output this cycle I1122 02:44:36.412275 3798194 cpu.go:275] no items to output this cycle E1122 02:44:46.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:44:46.411991 3798194 memory.go:184] no items to output this cycle I1122 02:44:46.412154 3798194 cpu.go:275] no items to output this cycle E1122 02:44:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:44:56.411964 3798194 memory.go:191] Add success. W1122 02:44:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:44:56.412011 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:44:56.412014 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:44:56.412363 3798194 cpu.go:282] Add success. I1122 02:44:56.441804 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:44:56.442424 3798194 net.go:642] Add success. I1122 02:44:56.446412 3798194 net.go:778] primary dev: ETH0 I1122 02:44:56.446426 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:44:56.446439 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:44:57.450280 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:44:57.450491 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:44:57.450504 3798194 disk_worker.go:723] disk space is not compliant W1122 02:44:57.450507 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:44:57.458551 3798194 disk_worker.go:509] system disk:vda1 I1122 02:44:57.458594 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:44:58.446493 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:44:59.450595 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:44:59.450673 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:44:59.450700 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:44:59.471234 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:45:01.494233 3798194 disk_info.go:125] begin check local disk info of client I1122 02:45:01.498087 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:45:01.498096 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000403040 0xc000403080] E1122 02:45:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:45:06.412253 3798194 memory.go:184] no items to output this cycle I1122 02:45:06.412406 3798194 cpu.go:275] no items to output this cycle E1122 02:45:16.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:45:16.412275 3798194 memory.go:184] no items to output this cycle I1122 02:45:16.412422 3798194 cpu.go:275] no items to output this cycle I1122 02:45:17.611224 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:45:17.611233 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:45:26.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:45:26.412893 3798194 memory.go:191] Add success. I1122 02:45:26.417446 3798194 cpu.go:282] Add success. I1122 02:45:26.441979 3798194 net.go:642] Add success. I1122 02:45:26.443435 3798194 net.go:778] primary dev: ETH0 I1122 02:45:26.443449 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:45:26.443461 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:45:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:45:29.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:45:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:45:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:45:36.411977 3798194 memory.go:184] no items to output this cycle I1122 02:45:36.412129 3798194 cpu.go:275] no items to output this cycle E1122 02:45:46.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:45:46.412271 3798194 memory.go:184] no items to output this cycle I1122 02:45:46.412420 3798194 cpu.go:275] no items to output this cycle E1122 02:45:56.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:45:56.412267 3798194 memory.go:191] Add success. W1122 02:45:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:45:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:45:56.412316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:45:56.416998 3798194 cpu.go:282] Add success. I1122 02:45:56.444045 3798194 event_worker.go:152] Polling the log file for events... I1122 02:45:56.444278 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:45:56.444756 3798194 net.go:642] Add success. I1122 02:45:56.446231 3798194 net.go:778] primary dev: ETH0 I1122 02:45:56.446246 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:45:56.446261 3798194 net.go:659] Add eni metrics success, count: 7 E1122 02:45:57.450986 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 02:45:57.451002 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:45:57.451233 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:45:57.451243 3798194 disk_worker.go:723] disk space is not compliant W1122 02:45:57.451246 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:45:57.460077 3798194 disk_worker.go:509] system disk:vda1 I1122 02:45:57.460134 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 02:45:58.448073 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 02:45:58.448088 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 02:45:59.448611 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 02:45:59.449685 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:45:59.449735 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:45:59.449755 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:45:59.471256 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:46:01.498653 3798194 disk_info.go:125] begin check local disk info of client I1122 02:46:01.502182 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:46:01.502191 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003642c0 0xc000364300] E1122 02:46:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:46:06.411975 3798194 memory.go:184] no items to output this cycle I1122 02:46:06.412133 3798194 cpu.go:275] no items to output this cycle E1122 02:46:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:46:16.411976 3798194 memory.go:184] no items to output this cycle I1122 02:46:16.412130 3798194 cpu.go:275] no items to output this cycle E1122 02:46:26.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:46:26.412889 3798194 memory.go:191] Add success. I1122 02:46:26.417085 3798194 cpu.go:282] Add success. I1122 02:46:26.441644 3798194 net.go:642] Add success. I1122 02:46:26.443192 3798194 net.go:778] primary dev: ETH0 I1122 02:46:26.443207 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:46:26.443220 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:46:29.450406 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:46:29.450501 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:46:29.450529 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:46:36.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:46:36.412262 3798194 memory.go:184] no items to output this cycle I1122 02:46:36.412617 3798194 cpu.go:275] no items to output this cycle E1122 02:46:46.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:46:46.412888 3798194 memory.go:184] no items to output this cycle I1122 02:46:46.413030 3798194 cpu.go:275] no items to output this cycle E1122 02:46:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:46:56.412268 3798194 memory.go:191] Add success. W1122 02:46:56.412301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:46:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:46:56.412317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:46:56.416904 3798194 cpu.go:282] Add success. I1122 02:46:56.441307 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:46:56.441882 3798194 net.go:642] Add success. I1122 02:46:56.443516 3798194 net.go:778] primary dev: ETH0 I1122 02:46:56.443538 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:46:56.443551 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:46:57.450601 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:46:57.450812 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:46:57.450823 3798194 disk_worker.go:723] disk space is not compliant W1122 02:46:57.450826 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:46:57.459255 3798194 disk_worker.go:509] system disk:vda1 I1122 02:46:57.459299 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:46:58.446016 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:46:59.450031 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:46:59.450140 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:46:59.450171 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:46:59.470684 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:47:01.502270 3798194 disk_info.go:125] begin check local disk info of client I1122 02:47:01.505800 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:47:01.505809 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d8700 0xc0003d8740] E1122 02:47:06.411971 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:47:06.411995 3798194 memory.go:184] no items to output this cycle I1122 02:47:06.412154 3798194 cpu.go:275] no items to output this cycle E1122 02:47:16.413053 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:47:16.413075 3798194 memory.go:184] no items to output this cycle I1122 02:47:16.413237 3798194 cpu.go:275] no items to output this cycle E1122 02:47:26.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:47:26.412892 3798194 memory.go:191] Add success. I1122 02:47:26.416992 3798194 cpu.go:282] Add success. I1122 02:47:26.441513 3798194 net.go:642] Add success. I1122 02:47:26.442914 3798194 net.go:778] primary dev: ETH0 I1122 02:47:26.442929 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:47:26.442943 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:47:29.450044 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:47:29.450146 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:47:29.450173 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:47:36.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:47:36.411984 3798194 memory.go:184] no items to output this cycle I1122 02:47:36.412146 3798194 cpu.go:275] no items to output this cycle E1122 02:47:46.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:47:46.411991 3798194 memory.go:184] no items to output this cycle I1122 02:47:46.412148 3798194 cpu.go:275] no items to output this cycle E1122 02:47:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:47:56.411952 3798194 memory.go:191] Add success. W1122 02:47:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:47:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:47:56.411996 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:47:56.412334 3798194 cpu.go:282] Add success. I1122 02:47:56.441805 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:47:56.442783 3798194 net.go:642] Add success. I1122 02:47:56.447064 3798194 net.go:778] primary dev: ETH0 I1122 02:47:56.447079 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:47:56.447092 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:47:57.450985 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:47:57.451206 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:47:57.451217 3798194 disk_worker.go:723] disk space is not compliant W1122 02:47:57.451220 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:47:57.460055 3798194 disk_worker.go:509] system disk:vda1 I1122 02:47:57.460096 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:47:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:47:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:47:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:47:59.450132 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:47:59.471655 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:48:01.505886 3798194 disk_info.go:125] begin check local disk info of client I1122 02:48:01.509431 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:48:01.509440 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000070f00 0xc000070f40] E1122 02:48:06.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:48:06.412253 3798194 memory.go:184] no items to output this cycle I1122 02:48:06.412404 3798194 cpu.go:275] no items to output this cycle E1122 02:48:16.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:48:16.412880 3798194 memory.go:184] no items to output this cycle I1122 02:48:16.413026 3798194 cpu.go:275] no items to output this cycle I1122 02:48:17.613220 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:48:17.613229 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:48:26.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:48:26.412273 3798194 memory.go:191] Add success. I1122 02:48:26.416508 3798194 cpu.go:282] Add success. I1122 02:48:26.442105 3798194 net.go:642] Add success. I1122 02:48:26.443754 3798194 net.go:778] primary dev: ETH0 I1122 02:48:26.443768 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:48:26.443781 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:48:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:48:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:48:29.450144 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:48:36.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:48:36.411973 3798194 memory.go:184] no items to output this cycle I1122 02:48:36.412130 3798194 cpu.go:275] no items to output this cycle E1122 02:48:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:48:46.411977 3798194 memory.go:184] no items to output this cycle I1122 02:48:46.412135 3798194 cpu.go:275] no items to output this cycle E1122 02:48:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:48:56.411950 3798194 memory.go:191] Add success. W1122 02:48:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:48:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:48:56.411991 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:48:56.412345 3798194 cpu.go:282] Add success. I1122 02:48:56.441804 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:48:56.442383 3798194 net.go:642] Add success. I1122 02:48:56.447244 3798194 net.go:778] primary dev: ETH0 I1122 02:48:56.447259 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:48:56.447271 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:48:57.450742 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:48:57.450944 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:48:57.450955 3798194 disk_worker.go:723] disk space is not compliant W1122 02:48:57.450958 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:48:57.459096 3798194 disk_worker.go:509] system disk:vda1 I1122 02:48:57.459155 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:48:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:48:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:48:59.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:48:59.450132 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:48:59.470680 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:49:01.510320 3798194 disk_info.go:125] begin check local disk info of client I1122 02:49:01.514259 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:49:01.514270 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d8880 0xc0003d88c0] E1122 02:49:06.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:49:06.413260 3798194 memory.go:184] no items to output this cycle I1122 02:49:06.413514 3798194 cpu.go:275] no items to output this cycle E1122 02:49:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:49:16.411977 3798194 memory.go:184] no items to output this cycle I1122 02:49:16.412134 3798194 cpu.go:275] no items to output this cycle E1122 02:49:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:49:26.411955 3798194 memory.go:191] Add success. I1122 02:49:26.412326 3798194 cpu.go:282] Add success. I1122 02:49:26.441918 3798194 net.go:642] Add success. I1122 02:49:26.445970 3798194 net.go:778] primary dev: ETH0 I1122 02:49:26.445988 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:49:26.446002 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:49:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:49:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:49:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:49:36.412093 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:49:36.412133 3798194 memory.go:184] no items to output this cycle I1122 02:49:36.412292 3798194 cpu.go:275] no items to output this cycle E1122 02:49:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:49:46.411977 3798194 memory.go:184] no items to output this cycle I1122 02:49:46.412139 3798194 cpu.go:275] no items to output this cycle E1122 02:49:56.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:49:56.413272 3798194 memory.go:191] Add success. W1122 02:49:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:49:56.413316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:49:56.413320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:49:56.417825 3798194 cpu.go:282] Add success. I1122 02:49:56.441223 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:49:56.441686 3798194 net.go:642] Add success. I1122 02:49:56.443221 3798194 net.go:778] primary dev: ETH0 I1122 02:49:56.443235 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:49:56.443249 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:49:57.450989 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:49:57.451173 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:49:57.451231 3798194 disk_worker.go:723] disk space is not compliant W1122 02:49:57.451234 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:49:57.459935 3798194 disk_worker.go:509] system disk:vda1 I1122 02:49:57.459976 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:49:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:49:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:49:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:49:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:49:59.471623 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:50:01.514373 3798194 disk_info.go:125] begin check local disk info of client I1122 02:50:01.518287 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:50:01.518297 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036cf80 0xc00036cfc0] E1122 02:50:06.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:50:06.411972 3798194 memory.go:184] no items to output this cycle I1122 02:50:06.412118 3798194 cpu.go:275] no items to output this cycle E1122 02:50:16.413283 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:50:16.413313 3798194 memory.go:184] no items to output this cycle I1122 02:50:16.413529 3798194 cpu.go:275] no items to output this cycle E1122 02:50:26.412253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:50:26.412287 3798194 memory.go:191] Add success. I1122 02:50:26.417387 3798194 cpu.go:282] Add success. I1122 02:50:26.441937 3798194 net.go:642] Add success. I1122 02:50:26.443500 3798194 net.go:778] primary dev: ETH0 I1122 02:50:26.443515 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:50:26.443528 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:50:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:50:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:50:29.450511 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:50:36.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:50:36.413255 3798194 memory.go:184] no items to output this cycle I1122 02:50:36.413400 3798194 cpu.go:275] no items to output this cycle E1122 02:50:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:50:46.411983 3798194 memory.go:184] no items to output this cycle I1122 02:50:46.412146 3798194 cpu.go:275] no items to output this cycle E1122 02:50:56.413597 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:50:56.413629 3798194 memory.go:191] Add success. W1122 02:50:56.413664 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:50:56.413676 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:50:56.413679 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:50:56.417764 3798194 cpu.go:282] Add success. I1122 02:50:56.444516 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:50:56.444981 3798194 net.go:642] Add success. I1122 02:50:56.446519 3798194 net.go:778] primary dev: ETH0 I1122 02:50:56.446533 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:50:56.446546 3798194 net.go:659] Add eni metrics success, count: 7 E1122 02:50:57.450581 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 02:50:57.450595 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:50:57.450801 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:50:57.450810 3798194 disk_worker.go:723] disk space is not compliant W1122 02:50:57.450813 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:50:57.459201 3798194 disk_worker.go:509] system disk:vda1 I1122 02:50:57.459245 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 02:50:58.447801 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 02:50:58.447816 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 02:50:59.449388 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 02:50:59.450462 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:50:59.450513 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:50:59.450533 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:50:59.471040 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:51:01.519376 3798194 disk_info.go:125] begin check local disk info of client I1122 02:51:01.523576 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:51:01.523585 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f9300 0xc0003f9340] E1122 02:51:06.413151 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:51:06.413178 3798194 memory.go:184] no items to output this cycle I1122 02:51:06.413325 3798194 cpu.go:275] no items to output this cycle E1122 02:51:16.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:51:16.413268 3798194 memory.go:184] no items to output this cycle I1122 02:51:16.413418 3798194 cpu.go:275] no items to output this cycle I1122 02:51:17.619227 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:51:17.619236 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:51:26.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:51:26.412268 3798194 memory.go:191] Add success. I1122 02:51:26.416742 3798194 cpu.go:282] Add success. I1122 02:51:26.441230 3798194 net.go:642] Add success. I1122 02:51:26.442886 3798194 net.go:778] primary dev: ETH0 I1122 02:51:26.442901 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:51:26.442913 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:51:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:51:29.450490 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:51:29.450516 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:51:36.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:51:36.412876 3798194 memory.go:184] no items to output this cycle I1122 02:51:36.413019 3798194 cpu.go:275] no items to output this cycle E1122 02:51:46.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:51:46.412878 3798194 memory.go:184] no items to output this cycle I1122 02:51:46.413024 3798194 cpu.go:275] no items to output this cycle E1122 02:51:56.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:51:56.413277 3798194 memory.go:191] Add success. W1122 02:51:56.413310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:51:56.413321 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:51:56.413325 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:51:56.418086 3798194 cpu.go:282] Add success. I1122 02:51:56.441633 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:51:56.442630 3798194 net.go:642] Add success. I1122 02:51:56.444756 3798194 net.go:778] primary dev: ETH0 I1122 02:51:56.444771 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:51:56.444783 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:51:57.446288 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:51:57.446505 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:51:57.446578 3798194 disk_worker.go:723] disk space is not compliant W1122 02:51:57.446581 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:51:57.451669 3798194 disk_worker.go:509] system disk:vda1 I1122 02:51:57.451718 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:51:58.446374 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:51:59.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:51:59.450467 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:51:59.450493 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:51:59.471015 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:52:01.523683 3798194 disk_info.go:125] begin check local disk info of client I1122 02:52:01.527201 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:52:01.527209 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a4d80 0xc0004a4dc0] E1122 02:52:06.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:52:06.413258 3798194 memory.go:184] no items to output this cycle I1122 02:52:06.413406 3798194 cpu.go:275] no items to output this cycle E1122 02:52:16.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:52:16.413266 3798194 memory.go:184] no items to output this cycle I1122 02:52:16.413415 3798194 cpu.go:275] no items to output this cycle E1122 02:52:26.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:52:26.413271 3798194 memory.go:191] Add success. I1122 02:52:26.417701 3798194 cpu.go:282] Add success. I1122 02:52:26.442258 3798194 net.go:642] Add success. I1122 02:52:26.443949 3798194 net.go:778] primary dev: ETH0 I1122 02:52:26.443964 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:52:26.443977 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:52:29.450545 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:52:29.450628 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:52:29.450653 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:52:36.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:52:36.411982 3798194 memory.go:184] no items to output this cycle I1122 02:52:36.412144 3798194 cpu.go:275] no items to output this cycle E1122 02:52:46.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:52:46.412258 3798194 memory.go:184] no items to output this cycle I1122 02:52:46.412414 3798194 cpu.go:275] no items to output this cycle E1122 02:52:56.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:52:56.412888 3798194 memory.go:191] Add success. W1122 02:52:56.412922 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:52:56.412935 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:52:56.412938 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:52:56.417297 3798194 cpu.go:282] Add success. I1122 02:52:56.441667 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:52:56.442230 3798194 net.go:642] Add success. I1122 02:52:56.443653 3798194 net.go:778] primary dev: ETH0 I1122 02:52:56.443667 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:52:56.443680 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:52:57.451365 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:52:57.451533 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:52:57.451587 3798194 disk_worker.go:723] disk space is not compliant W1122 02:52:57.451591 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:52:57.459972 3798194 disk_worker.go:509] system disk:vda1 I1122 02:52:57.460014 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:52:58.446370 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:52:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:52:59.450124 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:52:59.450158 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:52:59.471699 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:53:01.527311 3798194 disk_info.go:125] begin check local disk info of client I1122 02:53:01.530863 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:53:01.530872 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f01c0 0xc0003f0200] E1122 02:53:06.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:53:06.412886 3798194 memory.go:184] no items to output this cycle I1122 02:53:06.413030 3798194 cpu.go:275] no items to output this cycle E1122 02:53:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:53:16.411979 3798194 memory.go:184] no items to output this cycle I1122 02:53:16.412146 3798194 cpu.go:275] no items to output this cycle E1122 02:53:26.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:53:26.412901 3798194 memory.go:191] Add success. I1122 02:53:26.417428 3798194 cpu.go:282] Add success. I1122 02:53:26.441956 3798194 net.go:642] Add success. I1122 02:53:26.443449 3798194 net.go:778] primary dev: ETH0 I1122 02:53:26.443464 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:53:26.443478 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:53:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:53:29.450135 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:53:29.450168 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:53:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:53:36.411981 3798194 memory.go:184] no items to output this cycle I1122 02:53:36.412133 3798194 cpu.go:275] no items to output this cycle E1122 02:53:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:53:46.411979 3798194 memory.go:184] no items to output this cycle I1122 02:53:46.412135 3798194 cpu.go:275] no items to output this cycle E1122 02:53:56.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:53:56.413269 3798194 memory.go:191] Add success. W1122 02:53:56.413302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:53:56.413314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:53:56.413317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:53:56.417549 3798194 cpu.go:282] Add success. I1122 02:53:56.440913 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:53:56.441445 3798194 net.go:642] Add success. I1122 02:53:56.443010 3798194 net.go:778] primary dev: ETH0 I1122 02:53:56.443025 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:53:56.443037 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:53:57.448980 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:53:57.449236 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:53:57.449247 3798194 disk_worker.go:723] disk space is not compliant W1122 02:53:57.449250 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:53:57.451056 3798194 disk_worker.go:509] system disk:vda1 I1122 02:53:57.451098 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:53:58.446010 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:53:59.450500 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:53:59.450586 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:53:59.450611 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:53:59.471093 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:54:01.531803 3798194 disk_info.go:125] begin check local disk info of client I1122 02:54:01.535706 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:54:01.535715 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00038f7c0 0xc00038f800] E1122 02:54:06.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:54:06.412266 3798194 memory.go:184] no items to output this cycle I1122 02:54:06.412409 3798194 cpu.go:275] no items to output this cycle E1122 02:54:16.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:54:16.411988 3798194 memory.go:184] no items to output this cycle I1122 02:54:16.412146 3798194 cpu.go:275] no items to output this cycle I1122 02:54:17.619438 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:54:17.619447 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:54:26.411343 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:54:26.411377 3798194 memory.go:191] Add success. I1122 02:54:26.411713 3798194 cpu.go:282] Add success. I1122 02:54:26.441680 3798194 net.go:642] Add success. I1122 02:54:26.446174 3798194 net.go:778] primary dev: ETH0 I1122 02:54:26.446190 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:54:26.446202 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:54:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:54:29.450124 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:54:29.450156 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:54:36.412343 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:54:36.412366 3798194 memory.go:184] no items to output this cycle I1122 02:54:36.412508 3798194 cpu.go:275] no items to output this cycle E1122 02:54:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:54:46.411969 3798194 memory.go:184] no items to output this cycle I1122 02:54:46.412120 3798194 cpu.go:275] no items to output this cycle E1122 02:54:56.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:54:56.411955 3798194 memory.go:191] Add success. W1122 02:54:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:54:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:54:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:54:56.412364 3798194 cpu.go:282] Add success. I1122 02:54:56.441822 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:54:56.442343 3798194 net.go:642] Add success. I1122 02:54:56.446090 3798194 net.go:778] primary dev: ETH0 I1122 02:54:56.446105 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:54:56.446128 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:54:57.450334 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:54:57.450553 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:54:57.450563 3798194 disk_worker.go:723] disk space is not compliant W1122 02:54:57.450566 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:54:57.458649 3798194 disk_worker.go:509] system disk:vda1 I1122 02:54:57.458691 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:54:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:54:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:54:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:54:59.450144 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:54:59.471667 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:55:01.536204 3798194 disk_info.go:125] begin check local disk info of client I1122 02:55:01.539744 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:55:01.539753 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040dbc0 0xc00040dc00] E1122 02:55:06.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:55:06.413265 3798194 memory.go:184] no items to output this cycle I1122 02:55:06.413414 3798194 cpu.go:275] no items to output this cycle E1122 02:55:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:55:16.411983 3798194 memory.go:184] no items to output this cycle I1122 02:55:16.412140 3798194 cpu.go:275] no items to output this cycle E1122 02:55:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:55:26.411953 3798194 memory.go:191] Add success. I1122 02:55:26.412293 3798194 cpu.go:282] Add success. I1122 02:55:26.441924 3798194 net.go:642] Add success. I1122 02:55:26.445608 3798194 net.go:778] primary dev: ETH0 I1122 02:55:26.445624 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:55:26.445638 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:55:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:55:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:55:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:55:36.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:55:36.413248 3798194 memory.go:184] no items to output this cycle I1122 02:55:36.413389 3798194 cpu.go:275] no items to output this cycle E1122 02:55:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:55:46.411976 3798194 memory.go:184] no items to output this cycle I1122 02:55:46.412130 3798194 cpu.go:275] no items to output this cycle E1122 02:55:56.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:55:56.412999 3798194 memory.go:191] Add success. W1122 02:55:56.413029 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:55:56.413042 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:55:56.413045 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:55:56.418254 3798194 cpu.go:282] Add success. I1122 02:55:56.441561 3798194 event_worker.go:152] Polling the log file for events... I1122 02:55:56.443073 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:55:56.443569 3798194 net.go:642] Add success. I1122 02:55:56.446348 3798194 net.go:778] primary dev: ETH0 I1122 02:55:56.446364 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:55:56.446377 3798194 net.go:659] Add eni metrics success, count: 7 E1122 02:55:57.445258 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 02:55:57.445276 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:55:57.445542 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:55:57.445553 3798194 disk_worker.go:723] disk space is not compliant W1122 02:55:57.445556 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:55:57.448062 3798194 disk_worker.go:509] system disk:vda1 I1122 02:55:57.448149 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 02:55:58.448155 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 02:55:58.448170 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 02:55:59.449016 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 02:55:59.450091 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:55:59.450155 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:55:59.450175 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:55:59.471690 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:56:01.539807 3798194 disk_info.go:125] begin check local disk info of client I1122 02:56:01.543685 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:56:01.543694 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030f0c0 0xc00030f100] E1122 02:56:06.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:56:06.413249 3798194 memory.go:184] no items to output this cycle I1122 02:56:06.413391 3798194 cpu.go:275] no items to output this cycle E1122 02:56:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:56:16.411973 3798194 memory.go:184] no items to output this cycle I1122 02:56:16.412124 3798194 cpu.go:275] no items to output this cycle E1122 02:56:26.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:56:26.412889 3798194 memory.go:191] Add success. I1122 02:56:26.417469 3798194 cpu.go:282] Add success. I1122 02:56:26.442019 3798194 net.go:642] Add success. I1122 02:56:26.443550 3798194 net.go:778] primary dev: ETH0 I1122 02:56:26.443564 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:56:26.443577 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:56:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:56:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:56:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:56:36.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:56:36.412871 3798194 memory.go:184] no items to output this cycle I1122 02:56:36.413014 3798194 cpu.go:275] no items to output this cycle E1122 02:56:46.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:56:46.411989 3798194 memory.go:184] no items to output this cycle I1122 02:56:46.412145 3798194 cpu.go:275] no items to output this cycle E1122 02:56:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:56:56.411958 3798194 memory.go:191] Add success. W1122 02:56:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:56:56.412003 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:56:56.412015 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:56:56.412354 3798194 cpu.go:282] Add success. I1122 02:56:56.441838 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:56:56.442447 3798194 net.go:642] Add success. I1122 02:56:56.446993 3798194 net.go:778] primary dev: ETH0 I1122 02:56:56.447015 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:56:56.447031 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:56:57.452442 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:56:57.452660 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:56:57.452671 3798194 disk_worker.go:723] disk space is not compliant W1122 02:56:57.452674 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:56:57.462103 3798194 disk_worker.go:509] system disk:vda1 I1122 02:56:57.462165 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:56:58.446011 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:56:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:56:59.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:56:59.450152 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:56:59.471668 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:57:01.544512 3798194 disk_info.go:125] begin check local disk info of client I1122 02:57:01.548042 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:57:01.548052 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041c600 0xc00041c640] E1122 02:57:06.412766 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:57:06.412788 3798194 memory.go:184] no items to output this cycle I1122 02:57:06.412939 3798194 cpu.go:275] no items to output this cycle E1122 02:57:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:57:16.411971 3798194 memory.go:184] no items to output this cycle I1122 02:57:16.412128 3798194 cpu.go:275] no items to output this cycle I1122 02:57:17.620227 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 02:57:17.620235 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 02:57:26.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:57:26.412897 3798194 memory.go:191] Add success. I1122 02:57:26.417160 3798194 cpu.go:282] Add success. I1122 02:57:26.441692 3798194 net.go:642] Add success. I1122 02:57:26.443210 3798194 net.go:778] primary dev: ETH0 I1122 02:57:26.443233 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:57:26.443247 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:57:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:57:29.450125 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:57:29.450157 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:57:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:57:36.411981 3798194 memory.go:184] no items to output this cycle I1122 02:57:36.412135 3798194 cpu.go:275] no items to output this cycle E1122 02:57:46.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:57:46.411969 3798194 memory.go:184] no items to output this cycle I1122 02:57:46.412139 3798194 cpu.go:275] no items to output this cycle E1122 02:57:56.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:57:56.412889 3798194 memory.go:191] Add success. W1122 02:57:56.412922 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:57:56.412934 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:57:56.412938 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:57:56.417206 3798194 cpu.go:282] Add success. I1122 02:57:56.441603 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:57:56.442067 3798194 net.go:642] Add success. I1122 02:57:56.443645 3798194 net.go:778] primary dev: ETH0 I1122 02:57:56.443662 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:57:56.443675 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:57:57.451469 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:57:57.451633 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:57:57.451690 3798194 disk_worker.go:723] disk space is not compliant W1122 02:57:57.451693 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:57:57.462339 3798194 disk_worker.go:509] system disk:vda1 I1122 02:57:57.462395 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:57:58.446370 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:57:59.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:57:59.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:57:59.450511 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:57:59.471017 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:58:01.548884 3798194 disk_info.go:125] begin check local disk info of client I1122 02:58:01.552753 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:58:01.552763 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c0140 0xc0003c0180] E1122 02:58:06.412835 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:58:06.412863 3798194 memory.go:184] no items to output this cycle I1122 02:58:06.413004 3798194 cpu.go:275] no items to output this cycle E1122 02:58:16.414870 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:58:16.414897 3798194 memory.go:184] no items to output this cycle I1122 02:58:16.415043 3798194 cpu.go:275] no items to output this cycle E1122 02:58:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:58:26.412269 3798194 memory.go:191] Add success. I1122 02:58:26.417058 3798194 cpu.go:282] Add success. I1122 02:58:26.441871 3798194 net.go:642] Add success. I1122 02:58:26.443469 3798194 net.go:778] primary dev: ETH0 I1122 02:58:26.443485 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:58:26.443505 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:58:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:58:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:58:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:58:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:58:36.413264 3798194 memory.go:184] no items to output this cycle I1122 02:58:36.413410 3798194 cpu.go:275] no items to output this cycle E1122 02:58:46.411983 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:58:46.412008 3798194 memory.go:184] no items to output this cycle I1122 02:58:46.412180 3798194 cpu.go:275] no items to output this cycle E1122 02:58:56.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:58:56.412893 3798194 memory.go:191] Add success. W1122 02:58:56.412929 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:58:56.412940 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:58:56.412943 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:58:56.417374 3798194 cpu.go:282] Add success. I1122 02:58:56.441778 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:58:56.442294 3798194 net.go:642] Add success. I1122 02:58:56.443865 3798194 net.go:778] primary dev: ETH0 I1122 02:58:56.443891 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:58:56.443905 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:58:57.451423 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:58:57.451595 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:58:57.451650 3798194 disk_worker.go:723] disk space is not compliant W1122 02:58:57.451654 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:58:57.462139 3798194 disk_worker.go:509] system disk:vda1 I1122 02:58:57.462181 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:58:58.445991 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:58:59.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:58:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:58:59.450504 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:58:59.471038 3798194 disk_local_worker.go:436] Get disk info: [] I1122 02:59:01.552874 3798194 disk_info.go:125] begin check local disk info of client I1122 02:59:01.556727 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 02:59:01.556736 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00026d600 0xc00026d640] E1122 02:59:06.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:59:06.411984 3798194 memory.go:184] no items to output this cycle I1122 02:59:06.412143 3798194 cpu.go:275] no items to output this cycle E1122 02:59:16.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:59:16.413265 3798194 memory.go:184] no items to output this cycle I1122 02:59:16.413410 3798194 cpu.go:275] no items to output this cycle E1122 02:59:26.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:59:26.413258 3798194 memory.go:191] Add success. I1122 02:59:26.417643 3798194 cpu.go:282] Add success. I1122 02:59:26.442194 3798194 net.go:642] Add success. I1122 02:59:26.443745 3798194 net.go:778] primary dev: ETH0 I1122 02:59:26.443760 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:59:26.443774 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:59:29.450139 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:59:29.450219 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:59:29.450246 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 02:59:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:59:36.411977 3798194 memory.go:184] no items to output this cycle I1122 02:59:36.412125 3798194 cpu.go:275] no items to output this cycle E1122 02:59:46.413390 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:59:46.413443 3798194 memory.go:184] no items to output this cycle I1122 02:59:46.413650 3798194 cpu.go:275] no items to output this cycle E1122 02:59:56.411929 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 02:59:56.411965 3798194 memory.go:191] Add success. W1122 02:59:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 02:59:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 02:59:56.412008 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 02:59:56.412356 3798194 cpu.go:282] Add success. I1122 02:59:56.440955 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 02:59:56.441547 3798194 net.go:642] Add success. I1122 02:59:56.445583 3798194 net.go:778] primary dev: ETH0 I1122 02:59:56.445597 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 02:59:56.445610 3798194 net.go:659] Add eni metrics success, count: 7 I1122 02:59:57.451000 3798194 custom_config.go:106] query custom config with name: gpu W1122 02:59:57.451220 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 02:59:57.451231 3798194 disk_worker.go:723] disk space is not compliant W1122 02:59:57.451234 3798194 disk_worker.go:743] disk inode is not compliant I1122 02:59:57.460007 3798194 disk_worker.go:509] system disk:vda1 I1122 02:59:57.460050 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 02:59:58.446373 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 02:59:59.449971 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 02:59:59.450050 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 02:59:59.450074 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 02:59:59.471683 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:00:01.557146 3798194 disk_info.go:125] begin check local disk info of client I1122 03:00:01.560613 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:00:01.560622 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030eac0 0xc00030eb00] E1122 03:00:06.412982 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:00:06.413004 3798194 memory.go:184] no items to output this cycle I1122 03:00:06.413168 3798194 cpu.go:275] no items to output this cycle E1122 03:00:16.413296 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:00:16.413327 3798194 memory.go:184] no items to output this cycle I1122 03:00:16.413548 3798194 cpu.go:275] no items to output this cycle I1122 03:00:17.620391 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:00:17.620399 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:00:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:00:26.413264 3798194 memory.go:191] Add success. I1122 03:00:26.417406 3798194 cpu.go:282] Add success. I1122 03:00:26.441974 3798194 net.go:642] Add success. I1122 03:00:26.443424 3798194 net.go:778] primary dev: ETH0 I1122 03:00:26.443438 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:00:26.443452 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:00:29.450466 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:00:29.450541 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:00:29.450570 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:00:36.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:00:36.412249 3798194 memory.go:184] no items to output this cycle I1122 03:00:36.412390 3798194 cpu.go:275] no items to output this cycle E1122 03:00:46.411945 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:00:46.411972 3798194 memory.go:184] no items to output this cycle I1122 03:00:46.412131 3798194 cpu.go:275] no items to output this cycle E1122 03:00:56.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:00:56.412259 3798194 memory.go:191] Add success. W1122 03:00:56.412285 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:00:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:00:56.412300 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:00:56.416449 3798194 cpu.go:282] Add success. I1122 03:00:56.444744 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:00:56.445243 3798194 net.go:642] Add success. I1122 03:00:56.446758 3798194 net.go:778] primary dev: ETH0 I1122 03:00:56.446772 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:00:56.446785 3798194 net.go:659] Add eni metrics success, count: 7 E1122 03:00:57.450629 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 03:00:57.450642 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:00:57.450840 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:00:57.450850 3798194 disk_worker.go:723] disk space is not compliant W1122 03:00:57.450853 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:00:57.459198 3798194 disk_worker.go:509] system disk:vda1 I1122 03:00:57.459242 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 03:00:58.448091 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 03:00:58.448106 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 03:00:59.448616 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 03:00:59.449693 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:00:59.449753 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:00:59.449775 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:00:59.471301 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:01:01.562535 3798194 disk_info.go:125] begin check local disk info of client I1122 03:01:01.566445 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:01:01.566457 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00042cf40 0xc00042cf80] E1122 03:01:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:01:06.411978 3798194 memory.go:184] no items to output this cycle I1122 03:01:06.412134 3798194 cpu.go:275] no items to output this cycle E1122 03:01:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:01:16.413258 3798194 memory.go:184] no items to output this cycle I1122 03:01:16.413404 3798194 cpu.go:275] no items to output this cycle E1122 03:01:26.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:01:26.411968 3798194 memory.go:191] Add success. I1122 03:01:26.412315 3798194 cpu.go:282] Add success. I1122 03:01:26.441743 3798194 net.go:642] Add success. I1122 03:01:26.446313 3798194 net.go:778] primary dev: ETH0 I1122 03:01:26.446328 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:01:26.446341 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:01:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:01:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:01:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:01:36.412025 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:01:36.412051 3798194 memory.go:184] no items to output this cycle I1122 03:01:36.412216 3798194 cpu.go:275] no items to output this cycle E1122 03:01:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:01:46.411978 3798194 memory.go:184] no items to output this cycle I1122 03:01:46.412137 3798194 cpu.go:275] no items to output this cycle E1122 03:01:56.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:01:56.412273 3798194 memory.go:191] Add success. W1122 03:01:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:01:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:01:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:01:56.416586 3798194 cpu.go:282] Add success. I1122 03:01:56.440941 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:01:56.441531 3798194 net.go:642] Add success. I1122 03:01:56.443093 3798194 net.go:778] primary dev: ETH0 I1122 03:01:56.443119 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:01:56.443133 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:01:57.450932 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:01:57.451170 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:01:57.451182 3798194 disk_worker.go:723] disk space is not compliant W1122 03:01:57.451185 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:01:57.459993 3798194 disk_worker.go:509] system disk:vda1 I1122 03:01:57.460039 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:01:58.446375 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:01:59.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:01:59.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:01:59.450495 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:01:59.471005 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:02:01.566551 3798194 disk_info.go:125] begin check local disk info of client I1122 03:02:01.570393 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:02:01.570403 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ca240 0xc0003ca280] E1122 03:02:06.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:02:06.412877 3798194 memory.go:184] no items to output this cycle I1122 03:02:06.413017 3798194 cpu.go:275] no items to output this cycle E1122 03:02:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:02:16.413253 3798194 memory.go:184] no items to output this cycle I1122 03:02:16.413400 3798194 cpu.go:275] no items to output this cycle E1122 03:02:26.413306 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:02:26.413337 3798194 memory.go:191] Add success. I1122 03:02:26.418696 3798194 cpu.go:282] Add success. I1122 03:02:26.442280 3798194 net.go:642] Add success. I1122 03:02:26.444023 3798194 net.go:778] primary dev: ETH0 I1122 03:02:26.444038 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:02:26.444052 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:02:29.449942 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:02:29.450025 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:02:29.450055 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:02:36.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:02:36.412255 3798194 memory.go:184] no items to output this cycle I1122 03:02:36.412399 3798194 cpu.go:275] no items to output this cycle E1122 03:02:46.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:02:46.413257 3798194 memory.go:184] no items to output this cycle I1122 03:02:46.413400 3798194 cpu.go:275] no items to output this cycle E1122 03:02:56.413463 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:02:56.413498 3798194 memory.go:191] Add success. W1122 03:02:56.413526 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:02:56.413537 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:02:56.413540 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:02:56.417817 3798194 cpu.go:282] Add success. I1122 03:02:56.441572 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:02:56.442033 3798194 net.go:642] Add success. I1122 03:02:56.443590 3798194 net.go:778] primary dev: ETH0 I1122 03:02:56.443604 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:02:56.443618 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:02:57.450907 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:02:57.451104 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:02:57.451135 3798194 disk_worker.go:723] disk space is not compliant W1122 03:02:57.451138 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:02:57.460083 3798194 disk_worker.go:509] system disk:vda1 I1122 03:02:57.460144 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:02:58.445702 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:02:59.449667 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:02:59.449758 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:02:59.449784 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:02:59.471315 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:03:01.570508 3798194 disk_info.go:125] begin check local disk info of client I1122 03:03:01.574301 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:03:01.574309 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003cb040 0xc0003cb080] E1122 03:03:06.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:03:06.413259 3798194 memory.go:184] no items to output this cycle I1122 03:03:06.413403 3798194 cpu.go:275] no items to output this cycle E1122 03:03:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:03:16.413263 3798194 memory.go:184] no items to output this cycle I1122 03:03:16.413412 3798194 cpu.go:275] no items to output this cycle I1122 03:03:17.620560 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:03:17.620568 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:03:26.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:03:26.413269 3798194 memory.go:191] Add success. I1122 03:03:26.417768 3798194 cpu.go:282] Add success. I1122 03:03:26.441432 3798194 net.go:642] Add success. I1122 03:03:26.443099 3798194 net.go:778] primary dev: ETH0 I1122 03:03:26.443128 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:03:26.443143 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:03:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:03:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:03:29.450498 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:03:36.413735 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:03:36.413757 3798194 memory.go:184] no items to output this cycle I1122 03:03:36.413902 3798194 cpu.go:275] no items to output this cycle E1122 03:03:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:03:46.413260 3798194 memory.go:184] no items to output this cycle I1122 03:03:46.413414 3798194 cpu.go:275] no items to output this cycle E1122 03:03:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:03:56.413267 3798194 memory.go:191] Add success. W1122 03:03:56.413295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:03:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:03:56.413310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:03:56.417598 3798194 cpu.go:282] Add success. I1122 03:03:56.441105 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:03:56.441582 3798194 net.go:642] Add success. I1122 03:03:56.443162 3798194 net.go:778] primary dev: ETH0 I1122 03:03:56.443179 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:03:56.443204 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:03:57.450994 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:03:57.451232 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:03:57.451245 3798194 disk_worker.go:723] disk space is not compliant W1122 03:03:57.451248 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:03:57.460130 3798194 disk_worker.go:509] system disk:vda1 I1122 03:03:57.460174 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:03:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:03:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:03:59.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:03:59.450498 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:03:59.470954 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:04:01.574415 3798194 disk_info.go:125] begin check local disk info of client I1122 03:04:01.578378 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:04:01.578387 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004deb00 0xc0004deb40] E1122 03:04:06.412312 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:04:06.412335 3798194 memory.go:184] no items to output this cycle I1122 03:04:06.412477 3798194 cpu.go:275] no items to output this cycle E1122 03:04:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:04:16.412263 3798194 memory.go:184] no items to output this cycle I1122 03:04:16.412411 3798194 cpu.go:275] no items to output this cycle E1122 03:04:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:04:26.411954 3798194 memory.go:191] Add success. I1122 03:04:26.412320 3798194 cpu.go:282] Add success. I1122 03:04:26.442050 3798194 net.go:642] Add success. I1122 03:04:26.446742 3798194 net.go:778] primary dev: ETH0 I1122 03:04:26.446760 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:04:26.446773 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:04:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:04:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:04:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:04:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:04:36.412256 3798194 memory.go:184] no items to output this cycle I1122 03:04:36.412411 3798194 cpu.go:275] no items to output this cycle E1122 03:04:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:04:46.411982 3798194 memory.go:184] no items to output this cycle I1122 03:04:46.412147 3798194 cpu.go:275] no items to output this cycle E1122 03:04:56.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:04:56.412892 3798194 memory.go:191] Add success. W1122 03:04:56.412919 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:04:56.412930 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:04:56.412933 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:04:56.417160 3798194 cpu.go:282] Add success. I1122 03:04:56.441668 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:04:56.442269 3798194 net.go:642] Add success. I1122 03:04:56.443719 3798194 net.go:778] primary dev: ETH0 I1122 03:04:56.443737 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:04:56.443752 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:04:57.450565 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:04:57.450729 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:04:57.450781 3798194 disk_worker.go:723] disk space is not compliant W1122 03:04:57.450784 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:04:57.459055 3798194 disk_worker.go:509] system disk:vda1 I1122 03:04:57.459096 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:04:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:04:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:04:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:04:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:04:59.471648 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:05:01.578487 3798194 disk_info.go:125] begin check local disk info of client I1122 03:05:01.582327 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:05:01.582337 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041e040 0xc00041e080] E1122 03:05:06.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:05:06.413251 3798194 memory.go:184] no items to output this cycle I1122 03:05:06.413392 3798194 cpu.go:275] no items to output this cycle E1122 03:05:16.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:05:16.413258 3798194 memory.go:184] no items to output this cycle I1122 03:05:16.413530 3798194 cpu.go:275] no items to output this cycle E1122 03:05:26.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:05:26.412891 3798194 memory.go:191] Add success. I1122 03:05:26.417137 3798194 cpu.go:282] Add success. I1122 03:05:26.441630 3798194 net.go:642] Add success. I1122 03:05:26.443070 3798194 net.go:778] primary dev: ETH0 I1122 03:05:26.443084 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:05:26.443097 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:05:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:05:29.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:05:29.450148 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:05:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:05:36.413254 3798194 memory.go:184] no items to output this cycle I1122 03:05:36.413397 3798194 cpu.go:275] no items to output this cycle E1122 03:05:46.413292 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:05:46.413314 3798194 memory.go:184] no items to output this cycle I1122 03:05:46.413459 3798194 cpu.go:275] no items to output this cycle E1122 03:05:56.411909 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:05:56.411939 3798194 memory.go:191] Add success. W1122 03:05:56.411965 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:05:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:05:56.411980 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:05:56.412314 3798194 cpu.go:282] Add success. I1122 03:05:56.441553 3798194 event_worker.go:152] Polling the log file for events... I1122 03:05:56.444939 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:05:56.445514 3798194 net.go:642] Add success. I1122 03:05:56.449705 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795286156, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 03:05:56.449814 3798194 net.go:778] primary dev: ETH0 I1122 03:05:56.449830 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:05:56.449844 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:05:56.459903 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 03:05:57.446338 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 03:05:57.446352 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:05:57.446601 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:05:57.446612 3798194 disk_worker.go:723] disk space is not compliant W1122 03:05:57.446615 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:05:57.449419 3798194 disk_worker.go:509] system disk:vda1 I1122 03:05:57.449459 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 03:05:58.448538 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 03:05:58.448554 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 03:05:59.448992 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 03:05:59.450064 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:05:59.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:05:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:05:59.471654 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:06:01.582442 3798194 disk_info.go:125] begin check local disk info of client I1122 03:06:01.586249 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:06:01.586258 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00038c0c0 0xc00038c100] E1122 03:06:06.412957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:06:06.412980 3798194 memory.go:184] no items to output this cycle I1122 03:06:06.413145 3798194 cpu.go:275] no items to output this cycle E1122 03:06:16.412369 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:06:16.412393 3798194 memory.go:184] no items to output this cycle I1122 03:06:16.412536 3798194 cpu.go:275] no items to output this cycle I1122 03:06:17.624244 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:06:17.624254 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:06:26.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:06:26.412272 3798194 memory.go:191] Add success. I1122 03:06:26.416549 3798194 cpu.go:282] Add success. I1122 03:06:26.442097 3798194 net.go:642] Add success. I1122 03:06:26.443565 3798194 net.go:778] primary dev: ETH0 I1122 03:06:26.443581 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:06:26.443595 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:06:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:06:29.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:06:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:06:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:06:36.411977 3798194 memory.go:184] no items to output this cycle I1122 03:06:36.412144 3798194 cpu.go:275] no items to output this cycle E1122 03:06:46.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:06:46.411990 3798194 memory.go:184] no items to output this cycle I1122 03:06:46.412357 3798194 cpu.go:275] no items to output this cycle E1122 03:06:56.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:06:56.412263 3798194 memory.go:191] Add success. W1122 03:06:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:06:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:06:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:06:56.416734 3798194 cpu.go:282] Add success. I1122 03:06:56.441168 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:06:56.441764 3798194 net.go:642] Add success. I1122 03:06:56.443341 3798194 net.go:778] primary dev: ETH0 I1122 03:06:56.443356 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:06:56.443368 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:06:57.451379 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:06:57.451599 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:06:57.451609 3798194 disk_worker.go:723] disk space is not compliant W1122 03:06:57.451612 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:06:57.460046 3798194 disk_worker.go:509] system disk:vda1 I1122 03:06:57.460086 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:06:58.446374 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:06:59.450476 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:06:59.450566 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:06:59.450592 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:06:59.471128 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:07:01.586362 3798194 disk_info.go:125] begin check local disk info of client I1122 03:07:01.590236 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:07:01.590245 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030ba00 0xc00030ba80] E1122 03:07:06.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:07:06.412874 3798194 memory.go:184] no items to output this cycle I1122 03:07:06.413022 3798194 cpu.go:275] no items to output this cycle E1122 03:07:16.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:07:16.413273 3798194 memory.go:184] no items to output this cycle I1122 03:07:16.413435 3798194 cpu.go:275] no items to output this cycle E1122 03:07:26.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:07:26.412269 3798194 memory.go:191] Add success. I1122 03:07:26.416704 3798194 cpu.go:282] Add success. I1122 03:07:26.442375 3798194 net.go:642] Add success. I1122 03:07:26.443888 3798194 net.go:778] primary dev: ETH0 I1122 03:07:26.443902 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:07:26.443914 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:07:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:07:29.450086 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:07:29.450124 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:07:36.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:07:36.411972 3798194 memory.go:184] no items to output this cycle I1122 03:07:36.412120 3798194 cpu.go:275] no items to output this cycle E1122 03:07:46.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:07:46.413251 3798194 memory.go:184] no items to output this cycle I1122 03:07:46.413396 3798194 cpu.go:275] no items to output this cycle E1122 03:07:56.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:07:56.412279 3798194 memory.go:191] Add success. W1122 03:07:56.412323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:07:56.412335 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:07:56.412338 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:07:56.416656 3798194 cpu.go:282] Add success. I1122 03:07:56.440965 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:07:56.441578 3798194 net.go:642] Add success. I1122 03:07:56.443035 3798194 net.go:778] primary dev: ETH0 I1122 03:07:56.443050 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:07:56.443063 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:07:57.456128 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:07:57.456352 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:07:57.456364 3798194 disk_worker.go:723] disk space is not compliant W1122 03:07:57.456367 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:07:57.467040 3798194 disk_worker.go:509] system disk:vda1 I1122 03:07:57.467086 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:07:58.446381 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:07:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:07:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:07:59.450151 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:07:59.471671 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:08:01.590344 3798194 disk_info.go:125] begin check local disk info of client I1122 03:08:01.594189 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:08:01.594198 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a3140 0xc0004a3180] E1122 03:08:06.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:08:06.411992 3798194 memory.go:184] no items to output this cycle I1122 03:08:06.412141 3798194 cpu.go:275] no items to output this cycle E1122 03:08:16.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:08:16.413265 3798194 memory.go:184] no items to output this cycle I1122 03:08:16.413409 3798194 cpu.go:275] no items to output this cycle E1122 03:08:26.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:08:26.412263 3798194 memory.go:191] Add success. I1122 03:08:26.416934 3798194 cpu.go:282] Add success. I1122 03:08:26.441494 3798194 net.go:642] Add success. I1122 03:08:26.443187 3798194 net.go:778] primary dev: ETH0 I1122 03:08:26.443203 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:08:26.443215 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:08:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:08:29.450088 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:08:29.450125 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:08:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:08:36.412255 3798194 memory.go:184] no items to output this cycle I1122 03:08:36.412396 3798194 cpu.go:275] no items to output this cycle E1122 03:08:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:08:46.413255 3798194 memory.go:184] no items to output this cycle I1122 03:08:46.413411 3798194 cpu.go:275] no items to output this cycle E1122 03:08:56.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:08:56.412274 3798194 memory.go:191] Add success. W1122 03:08:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:08:56.412314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:08:56.412318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:08:56.416964 3798194 cpu.go:282] Add success. I1122 03:08:56.441359 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:08:56.441938 3798194 net.go:642] Add success. I1122 03:08:56.443691 3798194 net.go:778] primary dev: ETH0 I1122 03:08:56.443710 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:08:56.443723 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:08:57.451128 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:08:57.451328 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:08:57.451338 3798194 disk_worker.go:723] disk space is not compliant W1122 03:08:57.451341 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:08:57.460106 3798194 disk_worker.go:509] system disk:vda1 I1122 03:08:57.460164 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:08:58.446385 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:08:59.450482 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:08:59.450571 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:08:59.450601 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:08:59.471115 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:09:01.594294 3798194 disk_info.go:125] begin check local disk info of client I1122 03:09:01.598160 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:09:01.598169 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b2140 0xc0003b2180] E1122 03:09:06.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:09:06.412880 3798194 memory.go:184] no items to output this cycle I1122 03:09:06.413020 3798194 cpu.go:275] no items to output this cycle E1122 03:09:16.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:09:16.412258 3798194 memory.go:184] no items to output this cycle I1122 03:09:16.412405 3798194 cpu.go:275] no items to output this cycle I1122 03:09:17.625211 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:09:17.625221 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:09:26.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:09:26.412260 3798194 memory.go:191] Add success. I1122 03:09:26.416413 3798194 cpu.go:282] Add success. I1122 03:09:26.441973 3798194 net.go:642] Add success. I1122 03:09:26.443530 3798194 net.go:778] primary dev: ETH0 I1122 03:09:26.443546 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:09:26.443559 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:09:29.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:09:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:09:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:09:36.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:09:36.412248 3798194 memory.go:184] no items to output this cycle I1122 03:09:36.412403 3798194 cpu.go:275] no items to output this cycle E1122 03:09:46.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:09:46.413270 3798194 memory.go:184] no items to output this cycle I1122 03:09:46.413417 3798194 cpu.go:275] no items to output this cycle E1122 03:09:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:09:56.412269 3798194 memory.go:191] Add success. W1122 03:09:56.412295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:09:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:09:56.412309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:09:56.416602 3798194 cpu.go:282] Add success. I1122 03:09:56.440960 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:09:56.441558 3798194 net.go:642] Add success. I1122 03:09:56.443269 3798194 net.go:778] primary dev: ETH0 I1122 03:09:56.443285 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:09:56.443298 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:09:57.450558 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:09:57.450725 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:09:57.450776 3798194 disk_worker.go:723] disk space is not compliant W1122 03:09:57.450780 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:09:57.459166 3798194 disk_worker.go:509] system disk:vda1 I1122 03:09:57.459212 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:09:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:09:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:09:59.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:09:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:09:59.471641 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:10:01.598275 3798194 disk_info.go:125] begin check local disk info of client I1122 03:10:01.602248 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:10:01.602258 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b2e40 0xc0003b2e80] E1122 03:10:06.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:10:06.413261 3798194 memory.go:184] no items to output this cycle I1122 03:10:06.413399 3798194 cpu.go:275] no items to output this cycle E1122 03:10:16.412277 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:10:16.412305 3798194 memory.go:184] no items to output this cycle I1122 03:10:16.412517 3798194 cpu.go:275] no items to output this cycle E1122 03:10:26.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:10:26.413270 3798194 memory.go:191] Add success. I1122 03:10:26.417487 3798194 cpu.go:282] Add success. I1122 03:10:26.441934 3798194 net.go:642] Add success. I1122 03:10:26.443662 3798194 net.go:778] primary dev: ETH0 I1122 03:10:26.443677 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:10:26.443690 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:10:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:10:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:10:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:10:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:10:36.412257 3798194 memory.go:184] no items to output this cycle I1122 03:10:36.412406 3798194 cpu.go:275] no items to output this cycle E1122 03:10:46.413263 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:10:46.413287 3798194 memory.go:184] no items to output this cycle I1122 03:10:46.413435 3798194 cpu.go:275] no items to output this cycle E1122 03:10:56.411072 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:10:56.411105 3798194 memory.go:191] Add success. W1122 03:10:56.411152 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:10:56.411165 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:10:56.411170 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:10:56.415699 3798194 cpu.go:282] Add success. I1122 03:10:56.444010 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:10:56.444512 3798194 net.go:642] Add success. I1122 03:10:56.446218 3798194 net.go:778] primary dev: ETH0 I1122 03:10:56.446234 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:10:56.446247 3798194 net.go:659] Add eni metrics success, count: 7 E1122 03:10:57.451404 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 03:10:57.451421 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:10:57.451643 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:10:57.451653 3798194 disk_worker.go:723] disk space is not compliant W1122 03:10:57.451656 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:10:57.460153 3798194 disk_worker.go:509] system disk:vda1 I1122 03:10:57.460191 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 03:10:58.448107 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 03:10:58.448142 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 03:10:59.448639 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 03:10:59.449712 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:10:59.449763 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:10:59.449784 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:10:59.471311 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:11:01.602358 3798194 disk_info.go:125] begin check local disk info of client I1122 03:11:01.606192 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:11:01.606202 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004cdb80 0xc0004cdbc0] E1122 03:11:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:11:06.412257 3798194 memory.go:184] no items to output this cycle I1122 03:11:06.412404 3798194 cpu.go:275] no items to output this cycle E1122 03:11:16.411637 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:11:16.411659 3798194 memory.go:184] no items to output this cycle I1122 03:11:16.411804 3798194 cpu.go:275] no items to output this cycle E1122 03:11:26.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:11:26.412277 3798194 memory.go:191] Add success. I1122 03:11:26.416787 3798194 cpu.go:282] Add success. I1122 03:11:26.441307 3798194 net.go:642] Add success. I1122 03:11:26.442843 3798194 net.go:778] primary dev: ETH0 I1122 03:11:26.442858 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:11:26.442870 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:11:29.449965 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:11:29.450045 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:11:29.450070 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:11:36.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:11:36.413264 3798194 memory.go:184] no items to output this cycle I1122 03:11:36.413405 3798194 cpu.go:275] no items to output this cycle E1122 03:11:46.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:11:46.412875 3798194 memory.go:184] no items to output this cycle I1122 03:11:46.413023 3798194 cpu.go:275] no items to output this cycle E1122 03:11:56.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:11:56.412270 3798194 memory.go:191] Add success. W1122 03:11:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:11:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:11:56.412313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:11:56.416824 3798194 cpu.go:282] Add success. I1122 03:11:56.441271 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:11:56.441826 3798194 net.go:642] Add success. I1122 03:11:56.443805 3798194 net.go:778] primary dev: ETH0 I1122 03:11:56.443822 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:11:56.443834 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:11:57.450540 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:11:57.450736 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:11:57.450746 3798194 disk_worker.go:723] disk space is not compliant W1122 03:11:57.450749 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:11:57.459108 3798194 disk_worker.go:509] system disk:vda1 I1122 03:11:57.459165 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:11:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:11:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:11:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:11:59.450146 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:11:59.471666 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:12:01.606298 3798194 disk_info.go:125] begin check local disk info of client I1122 03:12:01.610143 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:12:01.610153 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000410e00 0xc000410e40] E1122 03:12:06.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:12:06.413251 3798194 memory.go:184] no items to output this cycle I1122 03:12:06.413392 3798194 cpu.go:275] no items to output this cycle E1122 03:12:16.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:12:16.412883 3798194 memory.go:184] no items to output this cycle I1122 03:12:16.413027 3798194 cpu.go:275] no items to output this cycle I1122 03:12:17.625388 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:12:17.625397 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:12:26.413268 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:12:26.413306 3798194 memory.go:191] Add success. I1122 03:12:26.417563 3798194 cpu.go:282] Add success. I1122 03:12:26.442125 3798194 net.go:642] Add success. I1122 03:12:26.443661 3798194 net.go:778] primary dev: ETH0 I1122 03:12:26.443676 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:12:26.443690 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:12:29.450006 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:12:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:12:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:12:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:12:36.412254 3798194 memory.go:184] no items to output this cycle I1122 03:12:36.412399 3798194 cpu.go:275] no items to output this cycle E1122 03:12:46.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:12:46.412874 3798194 memory.go:184] no items to output this cycle I1122 03:12:46.413023 3798194 cpu.go:275] no items to output this cycle E1122 03:12:56.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:12:56.412895 3798194 memory.go:191] Add success. W1122 03:12:56.412929 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:12:56.412940 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:12:56.412944 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:12:56.417070 3798194 cpu.go:282] Add success. I1122 03:12:56.441428 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:12:56.441913 3798194 net.go:642] Add success. I1122 03:12:56.443470 3798194 net.go:778] primary dev: ETH0 I1122 03:12:56.443485 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:12:56.443499 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:12:57.450784 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:12:57.450997 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:12:57.451009 3798194 disk_worker.go:723] disk space is not compliant W1122 03:12:57.451012 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:12:57.459782 3798194 disk_worker.go:509] system disk:vda1 I1122 03:12:57.459826 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:12:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:12:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:12:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:12:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:12:59.471687 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:13:01.610255 3798194 disk_info.go:125] begin check local disk info of client I1122 03:13:01.614046 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:13:01.614056 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004892c0 0xc000489300] E1122 03:13:06.412023 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:13:06.412050 3798194 memory.go:184] no items to output this cycle I1122 03:13:06.412212 3798194 cpu.go:275] no items to output this cycle E1122 03:13:16.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:13:16.412255 3798194 memory.go:184] no items to output this cycle I1122 03:13:16.412410 3798194 cpu.go:275] no items to output this cycle E1122 03:13:26.411992 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:13:26.412027 3798194 memory.go:191] Add success. I1122 03:13:26.416639 3798194 cpu.go:282] Add success. I1122 03:13:26.442091 3798194 net.go:642] Add success. I1122 03:13:26.443501 3798194 net.go:778] primary dev: ETH0 I1122 03:13:26.443516 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:13:26.443530 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:13:29.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:13:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:13:29.450124 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:13:36.411009 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:13:36.411034 3798194 memory.go:184] no items to output this cycle I1122 03:13:36.411228 3798194 cpu.go:275] no items to output this cycle E1122 03:13:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:13:46.412258 3798194 memory.go:184] no items to output this cycle I1122 03:13:46.412403 3798194 cpu.go:275] no items to output this cycle E1122 03:13:56.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:13:56.412278 3798194 memory.go:191] Add success. W1122 03:13:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:13:56.412320 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:13:56.412323 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:13:56.417076 3798194 cpu.go:282] Add success. I1122 03:13:56.441187 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:13:56.441749 3798194 net.go:642] Add success. I1122 03:13:56.443412 3798194 net.go:778] primary dev: ETH0 I1122 03:13:56.443427 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:13:56.443439 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:13:57.446497 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:13:57.446763 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:13:57.446775 3798194 disk_worker.go:723] disk space is not compliant W1122 03:13:57.446778 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:13:57.448883 3798194 disk_worker.go:509] system disk:vda1 I1122 03:13:57.448925 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:13:58.446348 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:13:59.450465 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:13:59.450540 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:13:59.450565 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:13:59.471083 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:14:01.614153 3798194 disk_info.go:125] begin check local disk info of client I1122 03:14:01.618040 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:14:01.618049 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e500 0xc00030e540] E1122 03:14:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:14:06.411980 3798194 memory.go:184] no items to output this cycle I1122 03:14:06.412131 3798194 cpu.go:275] no items to output this cycle E1122 03:14:16.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:14:16.413258 3798194 memory.go:184] no items to output this cycle I1122 03:14:16.413400 3798194 cpu.go:275] no items to output this cycle E1122 03:14:26.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:14:26.413277 3798194 memory.go:191] Add success. I1122 03:14:26.417982 3798194 cpu.go:282] Add success. I1122 03:14:26.441553 3798194 net.go:642] Add success. I1122 03:14:26.443187 3798194 net.go:778] primary dev: ETH0 I1122 03:14:26.443204 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:14:26.443217 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:14:29.450004 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:14:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:14:29.450126 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:14:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:14:36.412259 3798194 memory.go:184] no items to output this cycle I1122 03:14:36.412408 3798194 cpu.go:275] no items to output this cycle E1122 03:14:46.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:14:46.412871 3798194 memory.go:184] no items to output this cycle I1122 03:14:46.413029 3798194 cpu.go:275] no items to output this cycle E1122 03:14:56.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:14:56.412884 3798194 memory.go:191] Add success. W1122 03:14:56.412918 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:14:56.412930 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:14:56.412933 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:14:56.417222 3798194 cpu.go:282] Add success. I1122 03:14:56.441576 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:14:56.442043 3798194 net.go:642] Add success. I1122 03:14:56.443659 3798194 net.go:778] primary dev: ETH0 I1122 03:14:56.443673 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:14:56.443686 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:14:57.452319 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:14:57.452483 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:14:57.452538 3798194 disk_worker.go:723] disk space is not compliant W1122 03:14:57.452541 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:14:57.464433 3798194 disk_worker.go:509] system disk:vda1 I1122 03:14:57.464483 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:14:58.445979 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:14:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:14:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:14:59.450150 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:14:59.471642 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:15:01.618145 3798194 disk_info.go:125] begin check local disk info of client I1122 03:15:01.622063 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:15:01.622072 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000417c40 0xc000417c80] E1122 03:15:06.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:15:06.413257 3798194 memory.go:184] no items to output this cycle I1122 03:15:06.413590 3798194 cpu.go:275] no items to output this cycle E1122 03:15:16.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:15:16.412872 3798194 memory.go:184] no items to output this cycle I1122 03:15:16.413017 3798194 cpu.go:275] no items to output this cycle I1122 03:15:17.625552 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:15:17.625561 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:15:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:15:26.412267 3798194 memory.go:191] Add success. I1122 03:15:26.416699 3798194 cpu.go:282] Add success. I1122 03:15:26.442249 3798194 net.go:642] Add success. I1122 03:15:26.443651 3798194 net.go:778] primary dev: ETH0 I1122 03:15:26.443665 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:15:26.443677 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:15:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:15:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:15:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:15:36.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:15:36.411984 3798194 memory.go:184] no items to output this cycle I1122 03:15:36.412144 3798194 cpu.go:275] no items to output this cycle E1122 03:15:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:15:46.411969 3798194 memory.go:184] no items to output this cycle I1122 03:15:46.412126 3798194 cpu.go:275] no items to output this cycle E1122 03:15:56.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:15:56.413264 3798194 memory.go:191] Add success. W1122 03:15:56.413291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:15:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:15:56.413306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:15:56.419241 3798194 cpu.go:282] Add success. I1122 03:15:56.446167 3798194 event_worker.go:152] Polling the log file for events... I1122 03:15:56.446382 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:15:56.446924 3798194 net.go:642] Add success. I1122 03:15:56.448375 3798194 net.go:778] primary dev: ETH0 I1122 03:15:56.448389 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:15:56.448403 3798194 net.go:659] Add eni metrics success, count: 7 E1122 03:15:57.450538 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 03:15:57.450551 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:15:57.450712 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:15:57.450763 3798194 disk_worker.go:723] disk space is not compliant W1122 03:15:57.450767 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:15:57.459097 3798194 disk_worker.go:509] system disk:vda1 I1122 03:15:57.459153 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 03:15:58.448069 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 03:15:58.448083 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 03:15:59.448612 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 03:15:59.449685 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:15:59.449734 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:15:59.449753 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:15:59.471188 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:16:01.622157 3798194 disk_info.go:125] begin check local disk info of client I1122 03:16:01.626006 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:16:01.626015 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000373180 0xc0003731c0] E1122 03:16:06.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:16:06.412872 3798194 memory.go:184] no items to output this cycle I1122 03:16:06.413016 3798194 cpu.go:275] no items to output this cycle E1122 03:16:16.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:16:16.413251 3798194 memory.go:184] no items to output this cycle I1122 03:16:16.413399 3798194 cpu.go:275] no items to output this cycle E1122 03:16:26.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:16:26.412263 3798194 memory.go:191] Add success. I1122 03:16:26.416893 3798194 cpu.go:282] Add success. I1122 03:16:26.441439 3798194 net.go:642] Add success. I1122 03:16:26.442981 3798194 net.go:778] primary dev: ETH0 I1122 03:16:26.442996 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:16:26.443008 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:16:29.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:16:29.450467 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:16:29.450492 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:16:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:16:36.412253 3798194 memory.go:184] no items to output this cycle I1122 03:16:36.412529 3798194 cpu.go:275] no items to output this cycle E1122 03:16:46.411409 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:16:46.411431 3798194 memory.go:184] no items to output this cycle I1122 03:16:46.411581 3798194 cpu.go:275] no items to output this cycle E1122 03:16:56.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:16:56.412282 3798194 memory.go:191] Add success. W1122 03:16:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:16:56.412322 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:16:56.412325 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:16:56.417069 3798194 cpu.go:282] Add success. I1122 03:16:56.441425 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:16:56.441895 3798194 net.go:642] Add success. I1122 03:16:56.443625 3798194 net.go:778] primary dev: ETH0 I1122 03:16:56.443640 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:16:56.443653 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:16:57.450625 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:16:57.450820 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:16:57.450833 3798194 disk_worker.go:723] disk space is not compliant W1122 03:16:57.450836 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:16:57.459093 3798194 disk_worker.go:509] system disk:vda1 I1122 03:16:57.459158 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:16:58.446008 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:16:59.450122 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:16:59.450196 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:16:59.450223 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:16:59.470698 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:17:01.626146 3798194 disk_info.go:125] begin check local disk info of client I1122 03:17:01.629992 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:17:01.630001 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000439840 0xc000439880] E1122 03:17:06.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:17:06.413266 3798194 memory.go:184] no items to output this cycle I1122 03:17:06.413414 3798194 cpu.go:275] no items to output this cycle E1122 03:17:16.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:17:16.413250 3798194 memory.go:184] no items to output this cycle I1122 03:17:16.413396 3798194 cpu.go:275] no items to output this cycle E1122 03:17:26.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:17:26.412271 3798194 memory.go:191] Add success. I1122 03:17:26.416683 3798194 cpu.go:282] Add success. I1122 03:17:26.442301 3798194 net.go:642] Add success. I1122 03:17:26.443871 3798194 net.go:778] primary dev: ETH0 I1122 03:17:26.443885 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:17:26.443900 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:17:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:17:29.450126 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:17:29.450154 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:17:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:17:36.411975 3798194 memory.go:184] no items to output this cycle I1122 03:17:36.412128 3798194 cpu.go:275] no items to output this cycle E1122 03:17:46.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:17:46.413264 3798194 memory.go:184] no items to output this cycle I1122 03:17:46.413412 3798194 cpu.go:275] no items to output this cycle E1122 03:17:56.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:17:56.413266 3798194 memory.go:191] Add success. W1122 03:17:56.413294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:17:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:17:56.413309 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:17:56.417402 3798194 cpu.go:282] Add success. I1122 03:17:56.441785 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:17:56.442280 3798194 net.go:642] Add success. I1122 03:17:56.443855 3798194 net.go:778] primary dev: ETH0 I1122 03:17:56.443870 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:17:56.443882 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:17:57.451087 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:17:57.451514 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:17:57.451527 3798194 disk_worker.go:723] disk space is not compliant W1122 03:17:57.451530 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:17:57.462903 3798194 disk_worker.go:509] system disk:vda1 I1122 03:17:57.462944 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:17:58.446009 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:17:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:17:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:17:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:17:59.471651 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:18:01.630145 3798194 disk_info.go:125] begin check local disk info of client I1122 03:18:01.633921 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:18:01.633930 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004919c0 0xc000491a00] E1122 03:18:06.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:18:06.411972 3798194 memory.go:184] no items to output this cycle I1122 03:18:06.412136 3798194 cpu.go:275] no items to output this cycle E1122 03:18:16.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:18:16.412263 3798194 memory.go:184] no items to output this cycle I1122 03:18:16.412415 3798194 cpu.go:275] no items to output this cycle I1122 03:18:17.625718 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:18:17.625727 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:18:26.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:18:26.413271 3798194 memory.go:191] Add success. I1122 03:18:26.417448 3798194 cpu.go:282] Add success. I1122 03:18:26.441998 3798194 net.go:642] Add success. I1122 03:18:26.443698 3798194 net.go:778] primary dev: ETH0 I1122 03:18:26.443714 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:18:26.443726 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:18:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:18:29.450088 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:18:29.450125 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:18:36.413056 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:18:36.413085 3798194 memory.go:184] no items to output this cycle I1122 03:18:36.413256 3798194 cpu.go:275] no items to output this cycle E1122 03:18:46.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:18:46.411985 3798194 memory.go:184] no items to output this cycle I1122 03:18:46.412143 3798194 cpu.go:275] no items to output this cycle E1122 03:18:56.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:18:56.412279 3798194 memory.go:191] Add success. W1122 03:18:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:18:56.412319 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:18:56.412322 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:18:56.416590 3798194 cpu.go:282] Add success. I1122 03:18:56.440950 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:18:56.441551 3798194 net.go:642] Add success. I1122 03:18:56.443171 3798194 net.go:778] primary dev: ETH0 I1122 03:18:56.443187 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:18:56.443201 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:18:57.450918 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:18:57.451138 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:18:57.451150 3798194 disk_worker.go:723] disk space is not compliant W1122 03:18:57.451153 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:18:57.459909 3798194 disk_worker.go:509] system disk:vda1 I1122 03:18:57.459951 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:18:58.446218 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:18:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:18:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:18:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:18:59.471654 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:19:01.634136 3798194 disk_info.go:125] begin check local disk info of client I1122 03:19:01.638133 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:19:01.638143 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d8c00 0xc0003d8c40] E1122 03:19:06.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:19:06.412877 3798194 memory.go:184] no items to output this cycle I1122 03:19:06.413017 3798194 cpu.go:275] no items to output this cycle E1122 03:19:16.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:19:16.412251 3798194 memory.go:184] no items to output this cycle I1122 03:19:16.412401 3798194 cpu.go:275] no items to output this cycle E1122 03:19:26.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:19:26.412274 3798194 memory.go:191] Add success. I1122 03:19:26.419353 3798194 cpu.go:282] Add success. I1122 03:19:26.441574 3798194 net.go:642] Add success. I1122 03:19:26.443424 3798194 net.go:778] primary dev: ETH0 I1122 03:19:26.443447 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:19:26.443468 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:19:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:19:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:19:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:19:36.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:19:36.412260 3798194 memory.go:184] no items to output this cycle I1122 03:19:36.412403 3798194 cpu.go:275] no items to output this cycle E1122 03:19:46.413404 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:19:46.413425 3798194 memory.go:184] no items to output this cycle I1122 03:19:46.413583 3798194 cpu.go:275] no items to output this cycle E1122 03:19:56.413310 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:19:56.413342 3798194 memory.go:191] Add success. W1122 03:19:56.413369 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:19:56.413380 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:19:56.413384 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:19:56.417879 3798194 cpu.go:282] Add success. I1122 03:19:56.441242 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:19:56.441707 3798194 net.go:642] Add success. I1122 03:19:56.443375 3798194 net.go:778] primary dev: ETH0 I1122 03:19:56.443389 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:19:56.443400 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:19:57.450799 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:19:57.450966 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:19:57.451019 3798194 disk_worker.go:723] disk space is not compliant W1122 03:19:57.451022 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:19:57.459766 3798194 disk_worker.go:509] system disk:vda1 I1122 03:19:57.459818 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:19:58.446374 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:19:59.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:19:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:19:59.450508 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:19:59.471017 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:20:01.639148 3798194 disk_info.go:125] begin check local disk info of client I1122 03:20:01.642958 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:20:01.642967 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000308800 0xc000308840] E1122 03:20:06.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:20:06.412876 3798194 memory.go:184] no items to output this cycle I1122 03:20:06.413022 3798194 cpu.go:275] no items to output this cycle E1122 03:20:16.413288 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:20:16.413321 3798194 memory.go:184] no items to output this cycle I1122 03:20:16.413530 3798194 cpu.go:275] no items to output this cycle E1122 03:20:26.412843 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:20:26.412878 3798194 memory.go:191] Add success. I1122 03:20:26.417300 3798194 cpu.go:282] Add success. I1122 03:20:26.441733 3798194 net.go:642] Add success. I1122 03:20:26.443292 3798194 net.go:778] primary dev: ETH0 I1122 03:20:26.443307 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:20:26.443321 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:20:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:20:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:20:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:20:36.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:20:36.411985 3798194 memory.go:184] no items to output this cycle I1122 03:20:36.412143 3798194 cpu.go:275] no items to output this cycle E1122 03:20:46.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:20:46.413256 3798194 memory.go:184] no items to output this cycle I1122 03:20:46.413402 3798194 cpu.go:275] no items to output this cycle E1122 03:20:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:20:56.411952 3798194 memory.go:191] Add success. W1122 03:20:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:20:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:20:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:20:56.412384 3798194 cpu.go:282] Add success. I1122 03:20:56.445385 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:20:56.445875 3798194 net.go:642] Add success. I1122 03:20:56.449839 3798194 net.go:778] primary dev: ETH0 I1122 03:20:56.449855 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:20:56.449868 3798194 net.go:659] Add eni metrics success, count: 7 E1122 03:20:57.450668 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 03:20:57.450683 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:20:57.450883 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:20:57.450892 3798194 disk_worker.go:723] disk space is not compliant W1122 03:20:57.450895 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:20:57.462788 3798194 disk_worker.go:509] system disk:vda1 I1122 03:20:57.462828 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 03:20:58.448081 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 03:20:58.448095 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 03:20:59.448617 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 03:20:59.449692 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:20:59.449744 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:20:59.449767 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:20:59.471270 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:21:01.643070 3798194 disk_info.go:125] begin check local disk info of client I1122 03:21:01.646898 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:21:01.646907 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e02c0 0xc0002e0300] E1122 03:21:06.412088 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:21:06.412127 3798194 memory.go:184] no items to output this cycle I1122 03:21:06.412272 3798194 cpu.go:275] no items to output this cycle E1122 03:21:16.413003 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:21:16.413030 3798194 memory.go:184] no items to output this cycle I1122 03:21:16.413210 3798194 cpu.go:275] no items to output this cycle I1122 03:21:17.625888 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:21:17.625896 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:21:26.411232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:21:26.411266 3798194 memory.go:191] Add success. I1122 03:21:26.411618 3798194 cpu.go:282] Add success. I1122 03:21:26.442199 3798194 net.go:642] Add success. I1122 03:21:26.445853 3798194 net.go:778] primary dev: ETH0 I1122 03:21:26.445869 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:21:26.445881 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:21:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:21:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:21:29.450126 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:21:36.412818 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:21:36.412840 3798194 memory.go:184] no items to output this cycle I1122 03:21:36.412991 3798194 cpu.go:275] no items to output this cycle E1122 03:21:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:21:46.412257 3798194 memory.go:184] no items to output this cycle I1122 03:21:46.412412 3798194 cpu.go:275] no items to output this cycle E1122 03:21:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:21:56.411947 3798194 memory.go:191] Add success. W1122 03:21:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:21:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:21:56.411991 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:21:56.412332 3798194 cpu.go:282] Add success. I1122 03:21:56.441800 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:21:56.442369 3798194 net.go:642] Add success. I1122 03:21:56.446522 3798194 net.go:778] primary dev: ETH0 I1122 03:21:56.446537 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:21:56.446554 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:21:57.451264 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:21:57.451581 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:21:57.451592 3798194 disk_worker.go:723] disk space is not compliant W1122 03:21:57.451596 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:21:57.461039 3798194 disk_worker.go:509] system disk:vda1 I1122 03:21:57.461080 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:21:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:21:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:21:59.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:21:59.450507 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:21:59.471021 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:22:01.648163 3798194 disk_info.go:125] begin check local disk info of client I1122 03:22:01.670512 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:22:01.670521 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fd9c0 0xc0003fda00] E1122 03:22:06.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:22:06.412257 3798194 memory.go:184] no items to output this cycle I1122 03:22:06.412402 3798194 cpu.go:275] no items to output this cycle E1122 03:22:16.411478 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:22:16.411503 3798194 memory.go:184] no items to output this cycle I1122 03:22:16.411687 3798194 cpu.go:275] no items to output this cycle E1122 03:22:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:22:26.413269 3798194 memory.go:191] Add success. I1122 03:22:26.417555 3798194 cpu.go:282] Add success. I1122 03:22:26.442086 3798194 net.go:642] Add success. I1122 03:22:26.443686 3798194 net.go:778] primary dev: ETH0 I1122 03:22:26.443700 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:22:26.443712 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:22:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:22:29.450127 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:22:29.450154 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:22:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:22:36.413261 3798194 memory.go:184] no items to output this cycle I1122 03:22:36.413416 3798194 cpu.go:275] no items to output this cycle E1122 03:22:46.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:22:46.413252 3798194 memory.go:184] no items to output this cycle I1122 03:22:46.413396 3798194 cpu.go:275] no items to output this cycle E1122 03:22:56.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:22:56.412886 3798194 memory.go:191] Add success. W1122 03:22:56.412922 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:22:56.412934 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:22:56.412937 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:22:56.417131 3798194 cpu.go:282] Add success. I1122 03:22:56.441419 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:22:56.441883 3798194 net.go:642] Add success. I1122 03:22:56.443723 3798194 net.go:778] primary dev: ETH0 I1122 03:22:56.443738 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:22:56.443752 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:22:57.451455 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:22:57.451617 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:22:57.451677 3798194 disk_worker.go:723] disk space is not compliant W1122 03:22:57.451681 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:22:57.461285 3798194 disk_worker.go:509] system disk:vda1 I1122 03:22:57.461329 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:22:58.446377 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:22:59.450476 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:22:59.450559 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:22:59.450585 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:22:59.471121 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:23:01.671136 3798194 disk_info.go:125] begin check local disk info of client I1122 03:23:01.674990 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:23:01.674999 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d8c40 0xc0003d8c80] E1122 03:23:06.412259 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:23:06.412285 3798194 memory.go:184] no items to output this cycle I1122 03:23:06.412429 3798194 cpu.go:275] no items to output this cycle E1122 03:23:16.413048 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:23:16.413074 3798194 memory.go:184] no items to output this cycle I1122 03:23:16.413238 3798194 cpu.go:275] no items to output this cycle E1122 03:23:26.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:23:26.412897 3798194 memory.go:191] Add success. I1122 03:23:26.417169 3798194 cpu.go:282] Add success. I1122 03:23:26.441700 3798194 net.go:642] Add success. I1122 03:23:26.443300 3798194 net.go:778] primary dev: ETH0 I1122 03:23:26.443315 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:23:26.443328 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:23:29.450010 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:23:29.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:23:29.450129 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:23:36.412865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:23:36.412888 3798194 memory.go:184] no items to output this cycle I1122 03:23:36.413035 3798194 cpu.go:275] no items to output this cycle E1122 03:23:46.413650 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:23:46.413671 3798194 memory.go:184] no items to output this cycle I1122 03:23:46.413826 3798194 cpu.go:275] no items to output this cycle E1122 03:23:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:23:56.411944 3798194 memory.go:191] Add success. W1122 03:23:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:23:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:23:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:23:56.412353 3798194 cpu.go:282] Add success. I1122 03:23:56.441834 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:23:56.442403 3798194 net.go:642] Add success. I1122 03:23:56.446316 3798194 net.go:778] primary dev: ETH0 I1122 03:23:56.446332 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:23:56.446345 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:23:57.451011 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:23:57.451245 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:23:57.451258 3798194 disk_worker.go:723] disk space is not compliant W1122 03:23:57.451261 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:23:57.461133 3798194 disk_worker.go:509] system disk:vda1 I1122 03:23:57.461172 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:23:58.446393 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:23:59.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:23:59.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:23:59.450512 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:23:59.471037 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:24:01.675146 3798194 disk_info.go:125] begin check local disk info of client I1122 03:24:01.679030 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:24:01.679039 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000269400 0xc000269440] E1122 03:24:06.413017 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:24:06.413041 3798194 memory.go:184] no items to output this cycle I1122 03:24:06.413201 3798194 cpu.go:275] no items to output this cycle E1122 03:24:16.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:24:16.411989 3798194 memory.go:184] no items to output this cycle I1122 03:24:16.412153 3798194 cpu.go:275] no items to output this cycle I1122 03:24:17.626059 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:24:17.626068 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:24:26.411320 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:24:26.411354 3798194 memory.go:191] Add success. I1122 03:24:26.411691 3798194 cpu.go:282] Add success. I1122 03:24:26.441299 3798194 net.go:642] Add success. I1122 03:24:26.445488 3798194 net.go:778] primary dev: ETH0 I1122 03:24:26.445504 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:24:26.445519 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:24:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:24:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:24:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:24:36.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:24:36.413268 3798194 memory.go:184] no items to output this cycle I1122 03:24:36.413408 3798194 cpu.go:275] no items to output this cycle E1122 03:24:46.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:24:46.412269 3798194 memory.go:184] no items to output this cycle I1122 03:24:46.412413 3798194 cpu.go:275] no items to output this cycle E1122 03:24:56.413223 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:24:56.413257 3798194 memory.go:191] Add success. W1122 03:24:56.413291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:24:56.413302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:24:56.413306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:24:56.417755 3798194 cpu.go:282] Add success. I1122 03:24:56.441193 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:24:56.441688 3798194 net.go:642] Add success. I1122 03:24:56.443128 3798194 net.go:778] primary dev: ETH0 I1122 03:24:56.443144 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:24:56.443157 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:24:57.450451 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:24:57.450615 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:24:57.450671 3798194 disk_worker.go:723] disk space is not compliant W1122 03:24:57.450674 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:24:57.462697 3798194 disk_worker.go:509] system disk:vda1 I1122 03:24:57.462743 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:24:58.445995 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:24:59.450010 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:24:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:24:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:24:59.471659 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:25:01.679461 3798194 disk_info.go:125] begin check local disk info of client I1122 03:25:01.683619 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:25:01.683628 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000248d80 0xc000248dc0] E1122 03:25:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:25:06.413255 3798194 memory.go:184] no items to output this cycle I1122 03:25:06.413399 3798194 cpu.go:275] no items to output this cycle E1122 03:25:16.412530 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:25:16.412553 3798194 memory.go:184] no items to output this cycle I1122 03:25:16.412700 3798194 cpu.go:275] no items to output this cycle E1122 03:25:26.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:25:26.411962 3798194 memory.go:191] Add success. I1122 03:25:26.412325 3798194 cpu.go:282] Add success. I1122 03:25:26.442052 3798194 net.go:642] Add success. I1122 03:25:26.446370 3798194 net.go:778] primary dev: ETH0 I1122 03:25:26.446386 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:25:26.446400 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:25:29.450008 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:25:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:25:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:25:36.412107 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:25:36.412150 3798194 memory.go:184] no items to output this cycle I1122 03:25:36.412301 3798194 cpu.go:275] no items to output this cycle E1122 03:25:46.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:25:46.411971 3798194 memory.go:184] no items to output this cycle I1122 03:25:46.412146 3798194 cpu.go:275] no items to output this cycle E1122 03:25:56.412223 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:25:56.412254 3798194 memory.go:191] Add success. W1122 03:25:56.412286 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:25:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:25:56.412300 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:25:56.416552 3798194 cpu.go:282] Add success. I1122 03:25:56.442195 3798194 event_worker.go:152] Polling the log file for events... I1122 03:25:56.442425 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:25:56.442914 3798194 net.go:642] Add success. I1122 03:25:56.445758 3798194 net.go:778] primary dev: ETH0 I1122 03:25:56.445776 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:25:56.445790 3798194 net.go:659] Add eni metrics success, count: 7 E1122 03:25:57.447209 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 03:25:57.447225 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:25:57.447434 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:25:57.447509 3798194 disk_worker.go:723] disk space is not compliant W1122 03:25:57.447512 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:25:57.451346 3798194 disk_worker.go:509] system disk:vda1 I1122 03:25:57.451386 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 03:25:58.448506 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 03:25:58.448522 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 03:25:59.448978 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 03:25:59.450053 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:25:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:25:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:25:59.471688 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:26:01.684325 3798194 disk_info.go:125] begin check local disk info of client I1122 03:26:01.688247 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:26:01.688256 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00049fec0 0xc00049ff00] E1122 03:26:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:26:06.412256 3798194 memory.go:184] no items to output this cycle I1122 03:26:06.412399 3798194 cpu.go:275] no items to output this cycle E1122 03:26:16.412871 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:26:16.412893 3798194 memory.go:184] no items to output this cycle I1122 03:26:16.413036 3798194 cpu.go:275] no items to output this cycle E1122 03:26:26.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:26:26.413276 3798194 memory.go:191] Add success. I1122 03:26:26.417880 3798194 cpu.go:282] Add success. I1122 03:26:26.441406 3798194 net.go:642] Add success. I1122 03:26:26.442861 3798194 net.go:778] primary dev: ETH0 I1122 03:26:26.442877 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:26:26.442891 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:26:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:26:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:26:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:26:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:26:36.412257 3798194 memory.go:184] no items to output this cycle I1122 03:26:36.412405 3798194 cpu.go:275] no items to output this cycle E1122 03:26:46.412913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:26:46.412936 3798194 memory.go:184] no items to output this cycle I1122 03:26:46.413083 3798194 cpu.go:275] no items to output this cycle E1122 03:26:56.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:26:56.412893 3798194 memory.go:191] Add success. W1122 03:26:56.412923 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:26:56.412935 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:26:56.412938 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:26:56.417183 3798194 cpu.go:282] Add success. I1122 03:26:56.441554 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:26:56.442037 3798194 net.go:642] Add success. I1122 03:26:56.443549 3798194 net.go:778] primary dev: ETH0 I1122 03:26:56.443564 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:26:56.443577 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:26:57.450728 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:26:57.450892 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:26:57.450944 3798194 disk_worker.go:723] disk space is not compliant W1122 03:26:57.450947 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:26:57.459243 3798194 disk_worker.go:509] system disk:vda1 I1122 03:26:57.459285 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:26:58.445731 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:26:59.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:26:59.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:26:59.450509 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:26:59.471040 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:27:01.689335 3798194 disk_info.go:125] begin check local disk info of client I1122 03:27:01.693106 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:27:01.693135 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f1180 0xc0003f11c0] E1122 03:27:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:27:06.413254 3798194 memory.go:184] no items to output this cycle I1122 03:27:06.413408 3798194 cpu.go:275] no items to output this cycle E1122 03:27:16.413012 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:27:16.413037 3798194 memory.go:184] no items to output this cycle I1122 03:27:16.413204 3798194 cpu.go:275] no items to output this cycle I1122 03:27:17.626223 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:27:17.626231 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:27:26.412003 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:27:26.412036 3798194 memory.go:191] Add success. I1122 03:27:26.416488 3798194 cpu.go:282] Add success. I1122 03:27:26.441999 3798194 net.go:642] Add success. I1122 03:27:26.443452 3798194 net.go:778] primary dev: ETH0 I1122 03:27:26.443467 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:27:26.443480 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:27:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:27:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:27:29.450150 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:27:36.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:27:36.412262 3798194 memory.go:184] no items to output this cycle I1122 03:27:36.412410 3798194 cpu.go:275] no items to output this cycle E1122 03:27:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:27:46.413264 3798194 memory.go:184] no items to output this cycle I1122 03:27:46.413414 3798194 cpu.go:275] no items to output this cycle E1122 03:27:56.412770 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:27:56.412804 3798194 memory.go:191] Add success. W1122 03:27:56.412833 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:27:56.412846 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:27:56.412849 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:27:56.417280 3798194 cpu.go:282] Add success. I1122 03:27:56.441722 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:27:56.442248 3798194 net.go:642] Add success. I1122 03:27:56.443828 3798194 net.go:778] primary dev: ETH0 I1122 03:27:56.443842 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:27:56.443855 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:27:57.451235 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:27:57.451399 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:27:57.451450 3798194 disk_worker.go:723] disk space is not compliant W1122 03:27:57.451453 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:27:57.459749 3798194 disk_worker.go:509] system disk:vda1 I1122 03:27:57.459791 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:27:58.446386 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:27:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:27:59.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:27:59.450499 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:27:59.471014 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:28:01.694156 3798194 disk_info.go:125] begin check local disk info of client I1122 03:28:01.698055 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:28:01.698066 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037c340 0xc00037c380] E1122 03:28:06.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:28:06.411978 3798194 memory.go:184] no items to output this cycle I1122 03:28:06.412132 3798194 cpu.go:275] no items to output this cycle E1122 03:28:16.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:28:16.413269 3798194 memory.go:184] no items to output this cycle I1122 03:28:16.413414 3798194 cpu.go:275] no items to output this cycle E1122 03:28:26.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:28:26.411947 3798194 memory.go:191] Add success. I1122 03:28:26.412293 3798194 cpu.go:282] Add success. I1122 03:28:26.442028 3798194 net.go:642] Add success. I1122 03:28:26.446060 3798194 net.go:778] primary dev: ETH0 I1122 03:28:26.446076 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:28:26.446088 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:28:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:28:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:28:29.450512 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:28:36.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:28:36.411992 3798194 memory.go:184] no items to output this cycle I1122 03:28:36.412151 3798194 cpu.go:275] no items to output this cycle E1122 03:28:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:28:46.411978 3798194 memory.go:184] no items to output this cycle I1122 03:28:46.412354 3798194 cpu.go:275] no items to output this cycle E1122 03:28:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:28:56.412264 3798194 memory.go:191] Add success. W1122 03:28:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:28:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:28:56.412314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:28:56.417156 3798194 cpu.go:282] Add success. I1122 03:28:56.441677 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:28:56.442167 3798194 net.go:642] Add success. I1122 03:28:56.443679 3798194 net.go:778] primary dev: ETH0 I1122 03:28:56.443693 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:28:56.443707 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:28:57.451486 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:28:57.451661 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:28:57.451730 3798194 disk_worker.go:723] disk space is not compliant W1122 03:28:57.451733 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:28:57.461856 3798194 disk_worker.go:509] system disk:vda1 I1122 03:28:57.461897 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:28:58.446377 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:28:59.449668 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:28:59.449745 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:28:59.449771 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:28:59.471184 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:29:01.699143 3798194 disk_info.go:125] begin check local disk info of client I1122 03:29:01.703055 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:29:01.703065 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004524c0 0xc000452500] E1122 03:29:06.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:29:06.411986 3798194 memory.go:184] no items to output this cycle I1122 03:29:06.412145 3798194 cpu.go:275] no items to output this cycle E1122 03:29:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:29:16.411981 3798194 memory.go:184] no items to output this cycle I1122 03:29:16.412140 3798194 cpu.go:275] no items to output this cycle E1122 03:29:26.412846 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:29:26.412878 3798194 memory.go:191] Add success. I1122 03:29:26.417333 3798194 cpu.go:282] Add success. I1122 03:29:26.441742 3798194 net.go:642] Add success. I1122 03:29:26.443580 3798194 net.go:778] primary dev: ETH0 I1122 03:29:26.443594 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:29:26.443606 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:29:29.449677 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:29:29.449759 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:29:29.449792 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:29:36.412509 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:29:36.412531 3798194 memory.go:184] no items to output this cycle I1122 03:29:36.412699 3798194 cpu.go:275] no items to output this cycle E1122 03:29:46.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:29:46.412880 3798194 memory.go:184] no items to output this cycle I1122 03:29:46.413026 3798194 cpu.go:275] no items to output this cycle E1122 03:29:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:29:56.411953 3798194 memory.go:191] Add success. W1122 03:29:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:29:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:29:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:29:56.412360 3798194 cpu.go:282] Add success. I1122 03:29:56.441835 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:29:56.442466 3798194 net.go:642] Add success. I1122 03:29:56.453154 3798194 net.go:778] primary dev: ETH0 I1122 03:29:56.453169 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:29:56.453192 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:29:57.450581 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:29:57.450773 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:29:57.450785 3798194 disk_worker.go:723] disk space is not compliant W1122 03:29:57.450788 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:29:57.459062 3798194 disk_worker.go:509] system disk:vda1 I1122 03:29:57.459105 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:29:58.446011 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:29:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:29:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:29:59.450132 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:29:59.471656 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:30:01.704152 3798194 disk_info.go:125] begin check local disk info of client I1122 03:30:01.707989 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:30:01.707999 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047c540 0xc00047c580] E1122 03:30:06.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:30:06.413263 3798194 memory.go:184] no items to output this cycle I1122 03:30:06.413405 3798194 cpu.go:275] no items to output this cycle E1122 03:30:16.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:30:16.411997 3798194 memory.go:184] no items to output this cycle I1122 03:30:16.412168 3798194 cpu.go:275] no items to output this cycle I1122 03:30:17.626396 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:30:17.626405 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:30:26.412910 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:30:26.412960 3798194 memory.go:191] Add success. I1122 03:30:26.417617 3798194 cpu.go:282] Add success. I1122 03:30:26.442463 3798194 net.go:642] Add success. I1122 03:30:26.444158 3798194 net.go:778] primary dev: ETH0 I1122 03:30:26.444182 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:30:26.444201 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:30:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:30:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:30:29.450511 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:30:36.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:30:36.413271 3798194 memory.go:184] no items to output this cycle I1122 03:30:36.413418 3798194 cpu.go:275] no items to output this cycle E1122 03:30:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:30:46.411973 3798194 memory.go:184] no items to output this cycle I1122 03:30:46.412136 3798194 cpu.go:275] no items to output this cycle E1122 03:30:56.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:30:56.411944 3798194 memory.go:191] Add success. W1122 03:30:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:30:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:30:56.411991 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:30:56.412340 3798194 cpu.go:282] Add success. I1122 03:30:56.444977 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:30:56.445473 3798194 net.go:642] Add success. I1122 03:30:56.449750 3798194 net.go:778] primary dev: ETH0 I1122 03:30:56.449768 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:30:56.449781 3798194 net.go:659] Add eni metrics success, count: 7 E1122 03:30:57.450689 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 03:30:57.450704 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:30:57.450905 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:30:57.450915 3798194 disk_worker.go:723] disk space is not compliant W1122 03:30:57.450918 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:30:57.461710 3798194 disk_worker.go:509] system disk:vda1 I1122 03:30:57.461754 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 03:30:58.448097 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 03:30:58.448129 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 03:30:59.448631 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 03:30:59.449705 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:30:59.449756 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:30:59.449774 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:30:59.471299 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:31:01.708092 3798194 disk_info.go:125] begin check local disk info of client I1122 03:31:01.711883 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:31:01.711891 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037f980 0xc00037f9c0] E1122 03:31:06.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:31:06.413253 3798194 memory.go:184] no items to output this cycle I1122 03:31:06.413397 3798194 cpu.go:275] no items to output this cycle E1122 03:31:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:31:16.411973 3798194 memory.go:184] no items to output this cycle I1122 03:31:16.412126 3798194 cpu.go:275] no items to output this cycle E1122 03:31:26.413671 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:31:26.413704 3798194 memory.go:191] Add success. I1122 03:31:26.418037 3798194 cpu.go:282] Add success. I1122 03:31:26.441723 3798194 net.go:642] Add success. I1122 03:31:26.443354 3798194 net.go:778] primary dev: ETH0 I1122 03:31:26.443369 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:31:26.443382 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:31:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:31:29.450158 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:31:29.450257 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:31:36.413007 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:31:36.413031 3798194 memory.go:184] no items to output this cycle I1122 03:31:36.413199 3798194 cpu.go:275] no items to output this cycle E1122 03:31:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:31:46.411974 3798194 memory.go:184] no items to output this cycle I1122 03:31:46.412128 3798194 cpu.go:275] no items to output this cycle E1122 03:31:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:31:56.411956 3798194 memory.go:191] Add success. W1122 03:31:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:31:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:31:56.412006 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:31:56.416262 3798194 cpu.go:282] Add success. I1122 03:31:56.441807 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:31:56.442314 3798194 net.go:642] Add success. I1122 03:31:56.444023 3798194 net.go:778] primary dev: ETH0 I1122 03:31:56.444046 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:31:56.444060 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:31:57.450895 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:31:57.451071 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:31:57.451146 3798194 disk_worker.go:723] disk space is not compliant W1122 03:31:57.451149 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:31:57.459940 3798194 disk_worker.go:509] system disk:vda1 I1122 03:31:57.459993 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:31:58.446481 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:31:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:31:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:31:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:31:59.471634 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:32:01.712146 3798194 disk_info.go:125] begin check local disk info of client I1122 03:32:01.715940 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:32:01.715952 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ccf40 0xc0003ccf80] E1122 03:32:06.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:32:06.411980 3798194 memory.go:184] no items to output this cycle I1122 03:32:06.412130 3798194 cpu.go:275] no items to output this cycle E1122 03:32:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:32:16.411978 3798194 memory.go:184] no items to output this cycle I1122 03:32:16.412136 3798194 cpu.go:275] no items to output this cycle E1122 03:32:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:32:26.412279 3798194 memory.go:191] Add success. I1122 03:32:26.416615 3798194 cpu.go:282] Add success. I1122 03:32:26.441316 3798194 net.go:642] Add success. I1122 03:32:26.442934 3798194 net.go:778] primary dev: ETH0 I1122 03:32:26.442949 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:32:26.442962 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:32:29.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:32:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:32:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:32:36.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:32:36.411970 3798194 memory.go:184] no items to output this cycle I1122 03:32:36.412124 3798194 cpu.go:275] no items to output this cycle E1122 03:32:46.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:32:46.413248 3798194 memory.go:184] no items to output this cycle I1122 03:32:46.413390 3798194 cpu.go:275] no items to output this cycle E1122 03:32:56.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:32:56.412886 3798194 memory.go:191] Add success. W1122 03:32:56.412914 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:32:56.412926 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:32:56.412929 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:32:56.417689 3798194 cpu.go:282] Add success. I1122 03:32:56.441182 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:32:56.441669 3798194 net.go:642] Add success. I1122 03:32:56.443285 3798194 net.go:778] primary dev: ETH0 I1122 03:32:56.443299 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:32:56.443312 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:32:57.450603 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:32:57.451057 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:32:57.451069 3798194 disk_worker.go:723] disk space is not compliant W1122 03:32:57.451072 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:32:57.462390 3798194 disk_worker.go:509] system disk:vda1 I1122 03:32:57.462440 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:32:58.446571 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:32:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:32:59.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:32:59.450126 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:32:59.471656 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:33:01.716055 3798194 disk_info.go:125] begin check local disk info of client I1122 03:33:01.719908 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:33:01.719916 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004cd9c0 0xc0004cda00] E1122 03:33:06.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:33:06.413262 3798194 memory.go:184] no items to output this cycle I1122 03:33:06.413408 3798194 cpu.go:275] no items to output this cycle E1122 03:33:16.412538 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:33:16.412569 3798194 memory.go:184] no items to output this cycle I1122 03:33:16.412723 3798194 cpu.go:275] no items to output this cycle I1122 03:33:17.626565 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:33:17.626573 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:33:26.411416 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:33:26.411449 3798194 memory.go:191] Add success. I1122 03:33:26.411794 3798194 cpu.go:282] Add success. I1122 03:33:26.441539 3798194 net.go:642] Add success. I1122 03:33:26.445440 3798194 net.go:778] primary dev: ETH0 I1122 03:33:26.445458 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:33:26.445472 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:33:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:33:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:33:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:33:36.412877 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:33:36.412902 3798194 memory.go:184] no items to output this cycle I1122 03:33:36.413050 3798194 cpu.go:275] no items to output this cycle E1122 03:33:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:33:46.411971 3798194 memory.go:184] no items to output this cycle I1122 03:33:46.412121 3798194 cpu.go:275] no items to output this cycle E1122 03:33:56.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:33:56.412891 3798194 memory.go:191] Add success. W1122 03:33:56.412932 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:33:56.412943 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:33:56.412947 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:33:56.417199 3798194 cpu.go:282] Add success. I1122 03:33:56.441641 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:33:56.442207 3798194 net.go:642] Add success. I1122 03:33:56.443653 3798194 net.go:778] primary dev: ETH0 I1122 03:33:56.443667 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:33:56.443680 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:33:57.450303 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:33:57.450466 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:33:57.450523 3798194 disk_worker.go:723] disk space is not compliant W1122 03:33:57.450527 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:33:57.461281 3798194 disk_worker.go:509] system disk:vda1 I1122 03:33:57.461326 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:33:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:33:59.450029 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:33:59.450124 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:33:59.450151 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:33:59.471706 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:34:01.720140 3798194 disk_info.go:125] begin check local disk info of client I1122 03:34:01.724342 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:34:01.724351 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d6c40 0xc0003d6c80] E1122 03:34:06.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:34:06.412877 3798194 memory.go:184] no items to output this cycle I1122 03:34:06.413021 3798194 cpu.go:275] no items to output this cycle E1122 03:34:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:34:16.411975 3798194 memory.go:184] no items to output this cycle I1122 03:34:16.412129 3798194 cpu.go:275] no items to output this cycle E1122 03:34:26.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:34:26.412891 3798194 memory.go:191] Add success. I1122 03:34:26.417412 3798194 cpu.go:282] Add success. I1122 03:34:26.441646 3798194 net.go:642] Add success. I1122 03:34:26.443379 3798194 net.go:778] primary dev: ETH0 I1122 03:34:26.443394 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:34:26.443408 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:34:29.450383 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:34:29.450466 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:34:29.450494 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:34:36.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:34:36.413257 3798194 memory.go:184] no items to output this cycle I1122 03:34:36.413402 3798194 cpu.go:275] no items to output this cycle E1122 03:34:46.412224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:34:46.412247 3798194 memory.go:184] no items to output this cycle I1122 03:34:46.412394 3798194 cpu.go:275] no items to output this cycle E1122 03:34:56.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:34:56.411953 3798194 memory.go:191] Add success. W1122 03:34:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:34:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:34:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:34:56.412367 3798194 cpu.go:282] Add success. I1122 03:34:56.441787 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:34:56.442313 3798194 net.go:642] Add success. I1122 03:34:56.446234 3798194 net.go:778] primary dev: ETH0 I1122 03:34:56.446249 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:34:56.446261 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:34:57.450600 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:34:57.450815 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:34:57.450827 3798194 disk_worker.go:723] disk space is not compliant W1122 03:34:57.450830 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:34:57.459229 3798194 disk_worker.go:509] system disk:vda1 I1122 03:34:57.459270 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:34:58.445862 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:34:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:34:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:34:59.450125 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:34:59.471636 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:35:01.725177 3798194 disk_info.go:125] begin check local disk info of client I1122 03:35:01.728909 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:35:01.728918 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000308800 0xc000308840] E1122 03:35:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:35:06.411982 3798194 memory.go:184] no items to output this cycle I1122 03:35:06.412134 3798194 cpu.go:275] no items to output this cycle E1122 03:35:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:35:16.411983 3798194 memory.go:184] no items to output this cycle I1122 03:35:16.412140 3798194 cpu.go:275] no items to output this cycle E1122 03:35:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:35:26.411949 3798194 memory.go:191] Add success. I1122 03:35:26.412305 3798194 cpu.go:282] Add success. I1122 03:35:26.441765 3798194 net.go:642] Add success. I1122 03:35:26.446261 3798194 net.go:778] primary dev: ETH0 I1122 03:35:26.446275 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:35:26.446296 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:35:29.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:35:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:35:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:35:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:35:36.411977 3798194 memory.go:184] no items to output this cycle I1122 03:35:36.412129 3798194 cpu.go:275] no items to output this cycle E1122 03:35:46.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:35:46.413258 3798194 memory.go:184] no items to output this cycle I1122 03:35:46.413406 3798194 cpu.go:275] no items to output this cycle E1122 03:35:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:35:56.411950 3798194 memory.go:191] Add success. W1122 03:35:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:35:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:35:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:35:56.412604 3798194 cpu.go:282] Add success. I1122 03:35:56.441537 3798194 event_worker.go:152] Polling the log file for events... I1122 03:35:56.444837 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:35:56.445325 3798194 net.go:642] Add success. I1122 03:35:56.453436 3798194 net.go:778] primary dev: ETH0 I1122 03:35:56.453451 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:35:56.453465 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:35:56.453483 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795287956, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 03:35:56.460938 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 03:35:57.445548 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 03:35:57.445566 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:35:57.445808 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:35:57.445819 3798194 disk_worker.go:723] disk space is not compliant W1122 03:35:57.445822 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:35:57.450842 3798194 disk_worker.go:509] system disk:vda1 I1122 03:35:57.450882 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 03:35:58.448161 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 03:35:58.448177 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 03:35:59.448984 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 03:35:59.450059 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:35:59.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:35:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:35:59.471644 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:36:01.729020 3798194 disk_info.go:125] begin check local disk info of client I1122 03:36:01.733000 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:36:01.733010 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e440 0xc00030e480] E1122 03:36:06.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:36:06.412265 3798194 memory.go:184] no items to output this cycle I1122 03:36:06.412409 3798194 cpu.go:275] no items to output this cycle E1122 03:36:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:36:16.411971 3798194 memory.go:184] no items to output this cycle I1122 03:36:16.412130 3798194 cpu.go:275] no items to output this cycle I1122 03:36:17.626732 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:36:17.626741 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:36:26.411451 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:36:26.411485 3798194 memory.go:191] Add success. I1122 03:36:26.411827 3798194 cpu.go:282] Add success. I1122 03:36:26.441442 3798194 net.go:642] Add success. I1122 03:36:26.445503 3798194 net.go:778] primary dev: ETH0 I1122 03:36:26.445518 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:36:26.445531 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:36:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:36:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:36:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:36:36.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:36:36.413271 3798194 memory.go:184] no items to output this cycle I1122 03:36:36.413415 3798194 cpu.go:275] no items to output this cycle E1122 03:36:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:36:46.411973 3798194 memory.go:184] no items to output this cycle I1122 03:36:46.412134 3798194 cpu.go:275] no items to output this cycle E1122 03:36:56.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:36:56.412889 3798194 memory.go:191] Add success. W1122 03:36:56.412925 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:36:56.412938 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:36:56.412941 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:36:56.417070 3798194 cpu.go:282] Add success. I1122 03:36:56.441427 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:36:56.441905 3798194 net.go:642] Add success. I1122 03:36:56.443633 3798194 net.go:778] primary dev: ETH0 I1122 03:36:56.443649 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:36:56.443663 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:36:57.450927 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:36:57.451125 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:36:57.451186 3798194 disk_worker.go:723] disk space is not compliant W1122 03:36:57.451189 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:36:57.460201 3798194 disk_worker.go:509] system disk:vda1 I1122 03:36:57.460257 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:36:58.446687 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:36:59.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:36:59.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:36:59.450510 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:36:59.471030 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:37:01.734144 3798194 disk_info.go:125] begin check local disk info of client I1122 03:37:01.738061 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:37:01.738070 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004065c0 0xc000406600] E1122 03:37:06.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:37:06.413253 3798194 memory.go:184] no items to output this cycle I1122 03:37:06.413395 3798194 cpu.go:275] no items to output this cycle E1122 03:37:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:37:16.411976 3798194 memory.go:184] no items to output this cycle I1122 03:37:16.412139 3798194 cpu.go:275] no items to output this cycle E1122 03:37:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:37:26.411954 3798194 memory.go:191] Add success. I1122 03:37:26.412318 3798194 cpu.go:282] Add success. I1122 03:37:26.441773 3798194 net.go:642] Add success. I1122 03:37:26.446133 3798194 net.go:778] primary dev: ETH0 I1122 03:37:26.446149 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:37:26.446163 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:37:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:37:29.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:37:29.450150 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:37:36.412931 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:37:36.412955 3798194 memory.go:184] no items to output this cycle I1122 03:37:36.413104 3798194 cpu.go:275] no items to output this cycle E1122 03:37:46.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:37:46.413266 3798194 memory.go:184] no items to output this cycle I1122 03:37:46.413418 3798194 cpu.go:275] no items to output this cycle E1122 03:37:56.413087 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:37:56.413138 3798194 memory.go:191] Add success. W1122 03:37:56.413174 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:37:56.413185 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:37:56.413188 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:37:56.418048 3798194 cpu.go:282] Add success. I1122 03:37:56.441576 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:37:56.442101 3798194 net.go:642] Add success. I1122 03:37:56.443733 3798194 net.go:778] primary dev: ETH0 I1122 03:37:56.443753 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:37:56.443775 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:37:57.449645 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:37:57.449888 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:37:57.449899 3798194 disk_worker.go:723] disk space is not compliant W1122 03:37:57.449902 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:37:57.451766 3798194 disk_worker.go:509] system disk:vda1 I1122 03:37:57.451809 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:37:58.445988 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:37:59.450498 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:37:59.450584 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:37:59.450609 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:37:59.471137 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:38:01.739137 3798194 disk_info.go:125] begin check local disk info of client I1122 03:38:01.742983 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:38:01.742992 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d02c0 0xc0003d0300] E1122 03:38:06.413145 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:38:06.413167 3798194 memory.go:184] no items to output this cycle I1122 03:38:06.413315 3798194 cpu.go:275] no items to output this cycle E1122 03:38:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:38:16.412256 3798194 memory.go:184] no items to output this cycle I1122 03:38:16.412413 3798194 cpu.go:275] no items to output this cycle E1122 03:38:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:38:26.412267 3798194 memory.go:191] Add success. I1122 03:38:26.416555 3798194 cpu.go:282] Add success. I1122 03:38:26.441206 3798194 net.go:642] Add success. I1122 03:38:26.442748 3798194 net.go:778] primary dev: ETH0 I1122 03:38:26.442762 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:38:26.442774 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:38:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:38:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:38:29.450514 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:38:36.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:38:36.412260 3798194 memory.go:184] no items to output this cycle I1122 03:38:36.412414 3798194 cpu.go:275] no items to output this cycle E1122 03:38:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:38:46.411984 3798194 memory.go:184] no items to output this cycle I1122 03:38:46.412136 3798194 cpu.go:275] no items to output this cycle E1122 03:38:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:38:56.412267 3798194 memory.go:191] Add success. W1122 03:38:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:38:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:38:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:38:56.416708 3798194 cpu.go:282] Add success. I1122 03:38:56.441085 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:38:56.441635 3798194 net.go:642] Add success. I1122 03:38:56.443179 3798194 net.go:778] primary dev: ETH0 I1122 03:38:56.443193 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:38:56.443206 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:38:57.451495 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:38:57.451662 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:38:57.451716 3798194 disk_worker.go:723] disk space is not compliant W1122 03:38:57.451720 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:38:57.460237 3798194 disk_worker.go:509] system disk:vda1 I1122 03:38:57.460278 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:38:58.445995 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:38:59.450652 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:38:59.450750 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:38:59.450779 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:38:59.471329 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:39:01.743091 3798194 disk_info.go:125] begin check local disk info of client I1122 03:39:01.747009 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:39:01.747018 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00027e800 0xc00027e840] E1122 03:39:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:39:06.411974 3798194 memory.go:184] no items to output this cycle I1122 03:39:06.412126 3798194 cpu.go:275] no items to output this cycle E1122 03:39:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:39:16.411975 3798194 memory.go:184] no items to output this cycle I1122 03:39:16.412132 3798194 cpu.go:275] no items to output this cycle I1122 03:39:17.626904 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:39:17.626913 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:39:26.411227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:39:26.411260 3798194 memory.go:191] Add success. I1122 03:39:26.411614 3798194 cpu.go:282] Add success. I1122 03:39:26.442226 3798194 net.go:642] Add success. I1122 03:39:26.446093 3798194 net.go:778] primary dev: ETH0 I1122 03:39:26.446125 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:39:26.446140 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:39:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:39:29.450461 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:39:29.450486 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:39:36.412807 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:39:36.412830 3798194 memory.go:184] no items to output this cycle I1122 03:39:36.412974 3798194 cpu.go:275] no items to output this cycle E1122 03:39:46.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:39:46.413256 3798194 memory.go:184] no items to output this cycle I1122 03:39:46.413400 3798194 cpu.go:275] no items to output this cycle E1122 03:39:56.412029 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:39:56.412066 3798194 memory.go:191] Add success. W1122 03:39:56.412099 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:39:56.412126 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:39:56.412131 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:39:56.412727 3798194 cpu.go:282] Add success. I1122 03:39:56.440959 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:39:56.441555 3798194 net.go:642] Add success. I1122 03:39:56.445564 3798194 net.go:778] primary dev: ETH0 I1122 03:39:56.445578 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:39:56.445590 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:39:57.450636 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:39:57.450832 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:39:57.450843 3798194 disk_worker.go:723] disk space is not compliant W1122 03:39:57.450846 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:39:57.459203 3798194 disk_worker.go:509] system disk:vda1 I1122 03:39:57.459245 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:39:58.445992 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:39:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:39:59.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:39:59.450151 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:39:59.471686 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:40:01.748145 3798194 disk_info.go:125] begin check local disk info of client I1122 03:40:01.752093 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:40:01.752103 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e040 0xc00030e080] E1122 03:40:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:40:06.412254 3798194 memory.go:184] no items to output this cycle I1122 03:40:06.412401 3798194 cpu.go:275] no items to output this cycle E1122 03:40:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:40:16.411978 3798194 memory.go:184] no items to output this cycle I1122 03:40:16.412148 3798194 cpu.go:275] no items to output this cycle E1122 03:40:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:40:26.413270 3798194 memory.go:191] Add success. I1122 03:40:26.417711 3798194 cpu.go:282] Add success. I1122 03:40:26.441298 3798194 net.go:642] Add success. I1122 03:40:26.442644 3798194 net.go:778] primary dev: ETH0 I1122 03:40:26.442658 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:40:26.442670 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:40:29.450215 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:40:29.450300 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:40:29.450338 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:40:36.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:40:36.411986 3798194 memory.go:184] no items to output this cycle I1122 03:40:36.412143 3798194 cpu.go:275] no items to output this cycle E1122 03:40:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:40:46.411981 3798194 memory.go:184] no items to output this cycle I1122 03:40:46.412137 3798194 cpu.go:275] no items to output this cycle E1122 03:40:56.413375 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:40:56.413410 3798194 memory.go:191] Add success. W1122 03:40:56.413436 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:40:56.413448 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:40:56.413451 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:40:56.418449 3798194 cpu.go:282] Add success. I1122 03:40:56.443351 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:40:56.443827 3798194 net.go:642] Add success. I1122 03:40:56.446696 3798194 net.go:778] primary dev: ETH0 I1122 03:40:56.446712 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:40:56.446730 3798194 net.go:659] Add eni metrics success, count: 7 E1122 03:40:57.446464 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 03:40:57.446479 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:40:57.446672 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:40:57.446743 3798194 disk_worker.go:723] disk space is not compliant W1122 03:40:57.446747 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:40:57.449513 3798194 disk_worker.go:509] system disk:vda1 I1122 03:40:57.449552 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 03:40:58.448097 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 03:40:58.448130 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 03:40:59.448655 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 03:40:59.449733 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:40:59.449785 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:40:59.449810 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:40:59.471338 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:41:01.752225 3798194 disk_info.go:125] begin check local disk info of client I1122 03:41:01.756138 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:41:01.756147 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fa680 0xc0003fa6c0] E1122 03:41:06.412847 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:41:06.412870 3798194 memory.go:184] no items to output this cycle I1122 03:41:06.413017 3798194 cpu.go:275] no items to output this cycle E1122 03:41:16.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:41:16.412873 3798194 memory.go:184] no items to output this cycle I1122 03:41:16.413016 3798194 cpu.go:275] no items to output this cycle E1122 03:41:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:41:26.411953 3798194 memory.go:191] Add success. I1122 03:41:26.412307 3798194 cpu.go:282] Add success. I1122 03:41:26.441896 3798194 net.go:642] Add success. I1122 03:41:26.445950 3798194 net.go:778] primary dev: ETH0 I1122 03:41:26.445975 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:41:26.445988 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:41:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:41:29.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:41:29.450152 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:41:36.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:41:36.413263 3798194 memory.go:184] no items to output this cycle I1122 03:41:36.413419 3798194 cpu.go:275] no items to output this cycle E1122 03:41:46.412833 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:41:46.412856 3798194 memory.go:184] no items to output this cycle I1122 03:41:46.413001 3798194 cpu.go:275] no items to output this cycle E1122 03:41:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:41:56.411947 3798194 memory.go:191] Add success. W1122 03:41:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:41:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:41:56.411991 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:41:56.412353 3798194 cpu.go:282] Add success. I1122 03:41:56.441823 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:41:56.442355 3798194 net.go:642] Add success. I1122 03:41:56.446346 3798194 net.go:778] primary dev: ETH0 I1122 03:41:56.446361 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:41:56.446373 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:41:57.450530 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:41:57.450725 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:41:57.450736 3798194 disk_worker.go:723] disk space is not compliant W1122 03:41:57.450739 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:41:57.459072 3798194 disk_worker.go:509] system disk:vda1 I1122 03:41:57.459136 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:41:58.445996 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:41:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:41:59.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:41:59.450150 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:41:59.471668 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:42:01.756245 3798194 disk_info.go:125] begin check local disk info of client I1122 03:42:01.760155 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:42:01.760164 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000397940 0xc000397980] E1122 03:42:06.412085 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:42:06.412122 3798194 memory.go:184] no items to output this cycle I1122 03:42:06.412272 3798194 cpu.go:275] no items to output this cycle E1122 03:42:16.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:42:16.413251 3798194 memory.go:184] no items to output this cycle I1122 03:42:16.413397 3798194 cpu.go:275] no items to output this cycle I1122 03:42:17.627074 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:42:17.627082 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:42:26.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:42:26.412259 3798194 memory.go:191] Add success. I1122 03:42:26.416708 3798194 cpu.go:282] Add success. I1122 03:42:26.441651 3798194 net.go:642] Add success. I1122 03:42:26.443738 3798194 net.go:778] primary dev: ETH0 I1122 03:42:26.443754 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:42:26.443767 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:42:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:42:29.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:42:29.450514 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:42:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:42:36.412259 3798194 memory.go:184] no items to output this cycle I1122 03:42:36.412398 3798194 cpu.go:275] no items to output this cycle E1122 03:42:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:42:46.411978 3798194 memory.go:184] no items to output this cycle I1122 03:42:46.412135 3798194 cpu.go:275] no items to output this cycle E1122 03:42:56.413044 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:42:56.413078 3798194 memory.go:191] Add success. W1122 03:42:56.413106 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:42:56.413134 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:42:56.413137 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:42:56.417683 3798194 cpu.go:282] Add success. I1122 03:42:56.440953 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:42:56.441524 3798194 net.go:642] Add success. I1122 03:42:56.442942 3798194 net.go:778] primary dev: ETH0 I1122 03:42:56.442956 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:42:56.442970 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:42:57.451308 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:42:57.451475 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:42:57.451529 3798194 disk_worker.go:723] disk space is not compliant W1122 03:42:57.451532 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:42:57.459894 3798194 disk_worker.go:509] system disk:vda1 I1122 03:42:57.459935 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:42:58.446428 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:42:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:42:59.450492 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:42:59.450517 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:42:59.471034 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:43:01.760265 3798194 disk_info.go:125] begin check local disk info of client I1122 03:43:01.764165 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:43:01.764174 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000435d40 0xc000435d80] E1122 03:43:06.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:43:06.412874 3798194 memory.go:184] no items to output this cycle I1122 03:43:06.413028 3798194 cpu.go:275] no items to output this cycle E1122 03:43:16.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:43:16.412260 3798194 memory.go:184] no items to output this cycle I1122 03:43:16.412407 3798194 cpu.go:275] no items to output this cycle E1122 03:43:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:43:26.411954 3798194 memory.go:191] Add success. I1122 03:43:26.412306 3798194 cpu.go:282] Add success. I1122 03:43:26.441747 3798194 net.go:642] Add success. I1122 03:43:26.446467 3798194 net.go:778] primary dev: ETH0 I1122 03:43:26.446482 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:43:26.446498 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:43:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:43:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:43:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:43:36.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:43:36.413270 3798194 memory.go:184] no items to output this cycle I1122 03:43:36.413422 3798194 cpu.go:275] no items to output this cycle E1122 03:43:46.412774 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:43:46.412795 3798194 memory.go:184] no items to output this cycle I1122 03:43:46.412947 3798194 cpu.go:275] no items to output this cycle E1122 03:43:56.412724 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:43:56.412760 3798194 memory.go:191] Add success. W1122 03:43:56.412786 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:43:56.412797 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:43:56.412801 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:43:56.417494 3798194 cpu.go:282] Add success. I1122 03:43:56.441552 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:43:56.442584 3798194 net.go:642] Add success. I1122 03:43:56.444178 3798194 net.go:778] primary dev: ETH0 I1122 03:43:56.444195 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:43:56.444213 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:43:57.450599 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:43:57.450757 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:43:57.450809 3798194 disk_worker.go:723] disk space is not compliant W1122 03:43:57.450812 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:43:57.459135 3798194 disk_worker.go:509] system disk:vda1 I1122 03:43:57.459174 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:43:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:43:59.450117 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:43:59.450212 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:43:59.450239 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:43:59.470705 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:44:01.765146 3798194 disk_info.go:125] begin check local disk info of client I1122 03:44:01.768981 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:44:01.768990 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004e4240 0xc0004e4280] E1122 03:44:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:44:06.411984 3798194 memory.go:184] no items to output this cycle I1122 03:44:06.412147 3798194 cpu.go:275] no items to output this cycle E1122 03:44:16.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:44:16.412265 3798194 memory.go:184] no items to output this cycle I1122 03:44:16.412428 3798194 cpu.go:275] no items to output this cycle E1122 03:44:26.412882 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:44:26.412919 3798194 memory.go:191] Add success. I1122 03:44:26.417295 3798194 cpu.go:282] Add success. I1122 03:44:26.442034 3798194 net.go:642] Add success. I1122 03:44:26.443628 3798194 net.go:778] primary dev: ETH0 I1122 03:44:26.443644 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:44:26.443657 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:44:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:44:29.450121 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:44:29.450149 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:44:36.412011 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:44:36.412033 3798194 memory.go:184] no items to output this cycle I1122 03:44:36.412198 3798194 cpu.go:275] no items to output this cycle E1122 03:44:46.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:44:46.413259 3798194 memory.go:184] no items to output this cycle I1122 03:44:46.413403 3798194 cpu.go:275] no items to output this cycle E1122 03:44:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:44:56.411947 3798194 memory.go:191] Add success. W1122 03:44:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:44:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:44:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:44:56.412348 3798194 cpu.go:282] Add success. I1122 03:44:56.441793 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:44:56.442397 3798194 net.go:642] Add success. I1122 03:44:56.446440 3798194 net.go:778] primary dev: ETH0 I1122 03:44:56.446456 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:44:56.446467 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:44:57.450675 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:44:57.450841 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:44:57.450895 3798194 disk_worker.go:723] disk space is not compliant W1122 03:44:57.450898 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:44:57.461525 3798194 disk_worker.go:509] system disk:vda1 I1122 03:44:57.461569 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:44:58.445695 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:44:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:44:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:44:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:44:59.471667 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:45:01.769084 3798194 disk_info.go:125] begin check local disk info of client I1122 03:45:01.773785 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:45:01.773796 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036e000 0xc00036e040] E1122 03:45:06.412096 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:45:06.412137 3798194 memory.go:184] no items to output this cycle I1122 03:45:06.412289 3798194 cpu.go:275] no items to output this cycle E1122 03:45:16.413021 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:45:16.413043 3798194 memory.go:184] no items to output this cycle I1122 03:45:16.413210 3798194 cpu.go:275] no items to output this cycle I1122 03:45:17.627224 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:45:17.627233 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:45:26.413028 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:45:26.413063 3798194 memory.go:191] Add success. I1122 03:45:26.417704 3798194 cpu.go:282] Add success. I1122 03:45:26.441282 3798194 net.go:642] Add success. I1122 03:45:26.442806 3798194 net.go:778] primary dev: ETH0 I1122 03:45:26.442822 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:45:26.442835 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:45:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:45:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:45:29.450144 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:45:36.412845 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:45:36.412870 3798194 memory.go:184] no items to output this cycle I1122 03:45:36.413009 3798194 cpu.go:275] no items to output this cycle E1122 03:45:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:45:46.411978 3798194 memory.go:184] no items to output this cycle I1122 03:45:46.412137 3798194 cpu.go:275] no items to output this cycle E1122 03:45:56.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:45:56.411958 3798194 memory.go:191] Add success. W1122 03:45:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:45:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:45:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:45:56.412332 3798194 cpu.go:282] Add success. I1122 03:45:56.441561 3798194 event_worker.go:152] Polling the log file for events... I1122 03:45:56.444679 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:45:56.445196 3798194 net.go:642] Add success. I1122 03:45:56.449302 3798194 net.go:778] primary dev: ETH0 I1122 03:45:56.449317 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:45:56.449330 3798194 net.go:659] Add eni metrics success, count: 7 E1122 03:45:57.450624 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 03:45:57.450638 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:45:57.450788 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:45:57.450838 3798194 disk_worker.go:723] disk space is not compliant W1122 03:45:57.450841 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:45:57.459189 3798194 disk_worker.go:509] system disk:vda1 I1122 03:45:57.459232 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 03:45:58.447799 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 03:45:58.447814 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 03:45:59.448613 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 03:45:59.449691 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:45:59.449748 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:45:59.449768 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:45:59.471259 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:46:01.774137 3798194 disk_info.go:125] begin check local disk info of client I1122 03:46:01.777913 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:46:01.777922 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000373600 0xc000373640] E1122 03:46:06.413158 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:46:06.413179 3798194 memory.go:184] no items to output this cycle I1122 03:46:06.413327 3798194 cpu.go:275] no items to output this cycle E1122 03:46:16.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:46:16.413265 3798194 memory.go:184] no items to output this cycle I1122 03:46:16.413410 3798194 cpu.go:275] no items to output this cycle E1122 03:46:26.411939 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:46:26.411973 3798194 memory.go:191] Add success. I1122 03:46:26.412350 3798194 cpu.go:282] Add success. I1122 03:46:26.442152 3798194 net.go:642] Add success. I1122 03:46:26.446569 3798194 net.go:778] primary dev: ETH0 I1122 03:46:26.446586 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:46:26.446709 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:46:29.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:46:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:46:29.450152 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:46:36.412774 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:46:36.412798 3798194 memory.go:184] no items to output this cycle I1122 03:46:36.412948 3798194 cpu.go:275] no items to output this cycle E1122 03:46:46.412875 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:46:46.412897 3798194 memory.go:184] no items to output this cycle I1122 03:46:46.413054 3798194 cpu.go:275] no items to output this cycle E1122 03:46:56.412475 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:46:56.412509 3798194 memory.go:191] Add success. W1122 03:46:56.412542 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:46:56.412553 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:46:56.412557 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:46:56.417143 3798194 cpu.go:282] Add success. I1122 03:46:56.441714 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:46:56.442275 3798194 net.go:642] Add success. I1122 03:46:56.443688 3798194 net.go:778] primary dev: ETH0 I1122 03:46:56.443702 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:46:56.443714 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:46:57.451042 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:46:57.451293 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:46:57.451305 3798194 disk_worker.go:723] disk space is not compliant W1122 03:46:57.451309 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:46:57.460094 3798194 disk_worker.go:509] system disk:vda1 I1122 03:46:57.460156 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:46:58.445986 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:46:59.450413 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:46:59.450500 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:46:59.450526 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:46:59.471066 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:47:01.779143 3798194 disk_info.go:125] begin check local disk info of client I1122 03:47:01.782984 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:47:01.782994 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041fa00 0xc00041fa40] E1122 03:47:06.411238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:47:06.411263 3798194 memory.go:184] no items to output this cycle I1122 03:47:06.411404 3798194 cpu.go:275] no items to output this cycle E1122 03:47:16.413460 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:47:16.413482 3798194 memory.go:184] no items to output this cycle I1122 03:47:16.413630 3798194 cpu.go:275] no items to output this cycle E1122 03:47:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:47:26.411962 3798194 memory.go:191] Add success. I1122 03:47:26.412301 3798194 cpu.go:282] Add success. I1122 03:47:26.441780 3798194 net.go:642] Add success. I1122 03:47:26.446172 3798194 net.go:778] primary dev: ETH0 I1122 03:47:26.446189 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:47:26.446203 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:47:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:47:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:47:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:47:36.412181 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:47:36.412205 3798194 memory.go:184] no items to output this cycle I1122 03:47:36.412349 3798194 cpu.go:275] no items to output this cycle E1122 03:47:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:47:46.411975 3798194 memory.go:184] no items to output this cycle I1122 03:47:46.412140 3798194 cpu.go:275] no items to output this cycle E1122 03:47:56.413038 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:47:56.413070 3798194 memory.go:191] Add success. W1122 03:47:56.413103 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:47:56.413128 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:47:56.413131 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:47:56.417890 3798194 cpu.go:282] Add success. I1122 03:47:56.441932 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:47:56.442881 3798194 net.go:642] Add success. I1122 03:47:56.444943 3798194 net.go:778] primary dev: ETH0 I1122 03:47:56.444958 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:47:56.444970 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:47:57.449273 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:47:57.449464 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:47:57.449520 3798194 disk_worker.go:723] disk space is not compliant W1122 03:47:57.449524 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:47:57.451343 3798194 disk_worker.go:509] system disk:vda1 I1122 03:47:57.451383 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:47:58.446375 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:47:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:47:59.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:47:59.450498 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:47:59.471013 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:48:01.783095 3798194 disk_info.go:125] begin check local disk info of client I1122 03:48:01.786908 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:48:01.786917 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004bb980 0xc0004bb9c0] E1122 03:48:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:48:06.411987 3798194 memory.go:184] no items to output this cycle I1122 03:48:06.412165 3798194 cpu.go:275] no items to output this cycle E1122 03:48:16.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:48:16.411970 3798194 memory.go:184] no items to output this cycle I1122 03:48:16.412124 3798194 cpu.go:275] no items to output this cycle I1122 03:48:17.627395 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:48:17.627403 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:48:26.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:48:26.412886 3798194 memory.go:191] Add success. I1122 03:48:26.417800 3798194 cpu.go:282] Add success. I1122 03:48:26.441471 3798194 net.go:642] Add success. I1122 03:48:26.443119 3798194 net.go:778] primary dev: ETH0 I1122 03:48:26.443134 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:48:26.443147 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:48:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:48:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:48:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:48:36.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:48:36.413270 3798194 memory.go:184] no items to output this cycle I1122 03:48:36.413418 3798194 cpu.go:275] no items to output this cycle E1122 03:48:46.411866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:48:46.411888 3798194 memory.go:184] no items to output this cycle I1122 03:48:46.412031 3798194 cpu.go:275] no items to output this cycle E1122 03:48:56.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:48:56.411962 3798194 memory.go:191] Add success. W1122 03:48:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:48:56.412006 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:48:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:48:56.412365 3798194 cpu.go:282] Add success. I1122 03:48:56.441790 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:48:56.442383 3798194 net.go:642] Add success. I1122 03:48:56.446420 3798194 net.go:778] primary dev: ETH0 I1122 03:48:56.446434 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:48:56.446448 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:48:57.451041 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:48:57.451271 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:48:57.451284 3798194 disk_worker.go:723] disk space is not compliant W1122 03:48:57.451287 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:48:57.459763 3798194 disk_worker.go:509] system disk:vda1 I1122 03:48:57.459809 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:48:58.446615 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:48:59.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:48:59.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:48:59.450495 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:48:59.470985 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:49:01.787017 3798194 disk_info.go:125] begin check local disk info of client I1122 03:49:01.790909 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:49:01.790920 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004108c0 0xc000410900] E1122 03:49:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:49:06.413254 3798194 memory.go:184] no items to output this cycle I1122 03:49:06.413612 3798194 cpu.go:275] no items to output this cycle E1122 03:49:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:49:16.411985 3798194 memory.go:184] no items to output this cycle I1122 03:49:16.412142 3798194 cpu.go:275] no items to output this cycle E1122 03:49:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:49:26.411953 3798194 memory.go:191] Add success. I1122 03:49:26.412334 3798194 cpu.go:282] Add success. I1122 03:49:26.441889 3798194 net.go:642] Add success. I1122 03:49:26.446344 3798194 net.go:778] primary dev: ETH0 I1122 03:49:26.446360 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:49:26.446375 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:49:29.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:49:29.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:49:29.450150 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:49:36.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:49:36.412249 3798194 memory.go:184] no items to output this cycle I1122 03:49:36.412394 3798194 cpu.go:275] no items to output this cycle E1122 03:49:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:49:46.413263 3798194 memory.go:184] no items to output this cycle I1122 03:49:46.413411 3798194 cpu.go:275] no items to output this cycle E1122 03:49:56.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:49:56.412267 3798194 memory.go:191] Add success. W1122 03:49:56.412295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:49:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:49:56.412311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:49:56.416554 3798194 cpu.go:282] Add success. I1122 03:49:56.441085 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:49:56.441584 3798194 net.go:642] Add success. I1122 03:49:56.443138 3798194 net.go:778] primary dev: ETH0 I1122 03:49:56.443153 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:49:56.443167 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:49:57.450969 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:49:57.451156 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:49:57.451216 3798194 disk_worker.go:723] disk space is not compliant W1122 03:49:57.451220 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:49:57.460072 3798194 disk_worker.go:509] system disk:vda1 I1122 03:49:57.460127 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:49:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:49:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:49:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:49:59.450152 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:49:59.471669 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:50:01.792148 3798194 disk_info.go:125] begin check local disk info of client I1122 03:50:01.795873 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:50:01.795883 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037cfc0 0xc00037d000] E1122 03:50:06.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:50:06.412880 3798194 memory.go:184] no items to output this cycle I1122 03:50:06.413021 3798194 cpu.go:275] no items to output this cycle E1122 03:50:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:50:16.411974 3798194 memory.go:184] no items to output this cycle I1122 03:50:16.412130 3798194 cpu.go:275] no items to output this cycle E1122 03:50:26.411929 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:50:26.411966 3798194 memory.go:191] Add success. I1122 03:50:26.412303 3798194 cpu.go:282] Add success. I1122 03:50:26.441743 3798194 net.go:642] Add success. I1122 03:50:26.446012 3798194 net.go:778] primary dev: ETH0 I1122 03:50:26.446028 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:50:26.446041 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:50:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:50:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:50:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:50:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:50:36.413256 3798194 memory.go:184] no items to output this cycle I1122 03:50:36.413397 3798194 cpu.go:275] no items to output this cycle E1122 03:50:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:50:46.411976 3798194 memory.go:184] no items to output this cycle I1122 03:50:46.412136 3798194 cpu.go:275] no items to output this cycle E1122 03:50:56.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:50:56.412264 3798194 memory.go:191] Add success. W1122 03:50:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:50:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:50:56.412311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:50:56.416710 3798194 cpu.go:282] Add success. I1122 03:50:56.444070 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:50:56.444559 3798194 net.go:642] Add success. I1122 03:50:56.446168 3798194 net.go:778] primary dev: ETH0 I1122 03:50:56.446182 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:50:56.446194 3798194 net.go:659] Add eni metrics success, count: 7 E1122 03:50:57.450289 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 03:50:57.450302 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:50:57.450462 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:50:57.450514 3798194 disk_worker.go:723] disk space is not compliant W1122 03:50:57.450517 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:50:57.458937 3798194 disk_worker.go:509] system disk:vda1 I1122 03:50:57.458980 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 03:50:58.448087 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 03:50:58.448100 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 03:50:59.448627 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 03:50:59.449703 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:50:59.449755 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:50:59.449775 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:50:59.471297 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:51:01.797145 3798194 disk_info.go:125] begin check local disk info of client I1122 03:51:01.800967 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:51:01.800976 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041b040 0xc00041b080] E1122 03:51:06.412846 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:51:06.412871 3798194 memory.go:184] no items to output this cycle I1122 03:51:06.413021 3798194 cpu.go:275] no items to output this cycle E1122 03:51:16.412335 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:51:16.412357 3798194 memory.go:184] no items to output this cycle I1122 03:51:16.412509 3798194 cpu.go:275] no items to output this cycle I1122 03:51:17.627562 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:51:17.627572 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:51:26.412257 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:51:26.412291 3798194 memory.go:191] Add success. I1122 03:51:26.416575 3798194 cpu.go:282] Add success. I1122 03:51:26.441289 3798194 net.go:642] Add success. I1122 03:51:26.442851 3798194 net.go:778] primary dev: ETH0 I1122 03:51:26.442867 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:51:26.442881 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:51:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:51:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:51:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:51:36.412253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:51:36.412276 3798194 memory.go:184] no items to output this cycle I1122 03:51:36.412420 3798194 cpu.go:275] no items to output this cycle E1122 03:51:46.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:51:46.413261 3798194 memory.go:184] no items to output this cycle I1122 03:51:46.413404 3798194 cpu.go:275] no items to output this cycle E1122 03:51:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:51:56.411945 3798194 memory.go:191] Add success. W1122 03:51:56.411975 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:51:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:51:56.411990 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:51:56.412326 3798194 cpu.go:282] Add success. I1122 03:51:56.441147 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:51:56.442005 3798194 net.go:642] Add success. I1122 03:51:56.446263 3798194 net.go:778] primary dev: ETH0 I1122 03:51:56.446280 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:51:56.446295 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:51:57.450553 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:51:57.450717 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:51:57.450771 3798194 disk_worker.go:723] disk space is not compliant W1122 03:51:57.450774 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:51:57.459126 3798194 disk_worker.go:509] system disk:vda1 I1122 03:51:57.459168 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:51:58.446027 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:51:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:51:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:51:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:51:59.471707 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:52:01.802144 3798194 disk_info.go:125] begin check local disk info of client I1122 03:52:01.805983 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:52:01.805992 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004aea80 0xc0004aeac0] E1122 03:52:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:52:06.411983 3798194 memory.go:184] no items to output this cycle I1122 03:52:06.412139 3798194 cpu.go:275] no items to output this cycle E1122 03:52:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:52:16.413258 3798194 memory.go:184] no items to output this cycle I1122 03:52:16.413410 3798194 cpu.go:275] no items to output this cycle E1122 03:52:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:52:26.411952 3798194 memory.go:191] Add success. I1122 03:52:26.412304 3798194 cpu.go:282] Add success. I1122 03:52:26.442035 3798194 net.go:642] Add success. I1122 03:52:26.446528 3798194 net.go:778] primary dev: ETH0 I1122 03:52:26.446545 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:52:26.446559 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:52:29.450123 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:52:29.450205 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:52:29.450232 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:52:36.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:52:36.413260 3798194 memory.go:184] no items to output this cycle I1122 03:52:36.413406 3798194 cpu.go:275] no items to output this cycle E1122 03:52:46.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:52:46.413247 3798194 memory.go:184] no items to output this cycle I1122 03:52:46.413392 3798194 cpu.go:275] no items to output this cycle E1122 03:52:56.413178 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:52:56.413210 3798194 memory.go:191] Add success. W1122 03:52:56.413242 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:52:56.413254 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:52:56.413257 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:52:56.417576 3798194 cpu.go:282] Add success. I1122 03:52:56.441095 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:52:56.441692 3798194 net.go:642] Add success. I1122 03:52:56.443561 3798194 net.go:778] primary dev: ETH0 I1122 03:52:56.443576 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:52:56.443587 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:52:57.451150 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:52:57.451357 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:52:57.451368 3798194 disk_worker.go:723] disk space is not compliant W1122 03:52:57.451371 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:52:57.459780 3798194 disk_worker.go:509] system disk:vda1 I1122 03:52:57.459823 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:52:58.445953 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:52:59.450055 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:52:59.450147 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:52:59.450172 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:52:59.471688 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:53:01.807134 3798194 disk_info.go:125] begin check local disk info of client I1122 03:53:01.811044 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:53:01.811053 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fc880 0xc0003fc8c0] E1122 03:53:06.413037 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:53:06.413058 3798194 memory.go:184] no items to output this cycle I1122 03:53:06.413215 3798194 cpu.go:275] no items to output this cycle E1122 03:53:16.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:53:16.412280 3798194 memory.go:184] no items to output this cycle I1122 03:53:16.412431 3798194 cpu.go:275] no items to output this cycle E1122 03:53:26.413256 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:53:26.413295 3798194 memory.go:191] Add success. I1122 03:53:26.417598 3798194 cpu.go:282] Add success. I1122 03:53:26.441304 3798194 net.go:642] Add success. I1122 03:53:26.442905 3798194 net.go:778] primary dev: ETH0 I1122 03:53:26.442926 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:53:26.442947 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:53:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:53:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:53:29.450516 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:53:36.411806 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:53:36.411832 3798194 memory.go:184] no items to output this cycle I1122 03:53:36.411982 3798194 cpu.go:275] no items to output this cycle E1122 03:53:46.412808 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:53:46.412830 3798194 memory.go:184] no items to output this cycle I1122 03:53:46.412997 3798194 cpu.go:275] no items to output this cycle E1122 03:53:56.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:53:56.412902 3798194 memory.go:191] Add success. W1122 03:53:56.412938 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:53:56.412950 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:53:56.412953 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:53:56.417342 3798194 cpu.go:282] Add success. I1122 03:53:56.441748 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:53:56.442340 3798194 net.go:642] Add success. I1122 03:53:56.449212 3798194 net.go:778] primary dev: ETH0 I1122 03:53:56.449226 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:53:56.449241 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:53:57.452936 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:53:57.453107 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:53:57.453192 3798194 disk_worker.go:723] disk space is not compliant W1122 03:53:57.453195 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:53:57.467346 3798194 disk_worker.go:509] system disk:vda1 I1122 03:53:57.467390 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:53:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:53:59.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:53:59.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:53:59.450497 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:53:59.471022 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:54:01.812142 3798194 disk_info.go:125] begin check local disk info of client I1122 03:54:01.816066 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:54:01.816075 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000372340 0xc000372380] E1122 03:54:06.411368 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:54:06.411390 3798194 memory.go:184] no items to output this cycle I1122 03:54:06.411532 3798194 cpu.go:275] no items to output this cycle E1122 03:54:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:54:16.412257 3798194 memory.go:184] no items to output this cycle I1122 03:54:16.412406 3798194 cpu.go:275] no items to output this cycle I1122 03:54:17.628218 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:54:17.628227 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:54:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:54:26.413264 3798194 memory.go:191] Add success. I1122 03:54:26.417845 3798194 cpu.go:282] Add success. I1122 03:54:26.441480 3798194 net.go:642] Add success. I1122 03:54:26.442938 3798194 net.go:778] primary dev: ETH0 I1122 03:54:26.442953 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:54:26.442964 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:54:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:54:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:54:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:54:36.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:54:36.413258 3798194 memory.go:184] no items to output this cycle I1122 03:54:36.413402 3798194 cpu.go:275] no items to output this cycle E1122 03:54:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:54:46.413269 3798194 memory.go:184] no items to output this cycle I1122 03:54:46.413414 3798194 cpu.go:275] no items to output this cycle E1122 03:54:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:54:56.412269 3798194 memory.go:191] Add success. W1122 03:54:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:54:56.412313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:54:56.412317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:54:56.416566 3798194 cpu.go:282] Add success. I1122 03:54:56.440947 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:54:56.441523 3798194 net.go:642] Add success. I1122 03:54:56.443077 3798194 net.go:778] primary dev: ETH0 I1122 03:54:56.443091 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:54:56.443104 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:54:57.454990 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:54:57.455177 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:54:57.455235 3798194 disk_worker.go:723] disk space is not compliant W1122 03:54:57.455238 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:54:57.463956 3798194 disk_worker.go:509] system disk:vda1 I1122 03:54:57.463998 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:54:58.446372 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:54:59.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:54:59.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:54:59.450499 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:54:59.471007 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:55:01.816179 3798194 disk_info.go:125] begin check local disk info of client I1122 03:55:01.819959 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:55:01.819968 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041e580 0xc00041e5c0] E1122 03:55:06.412590 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:55:06.412611 3798194 memory.go:184] no items to output this cycle I1122 03:55:06.412768 3798194 cpu.go:275] no items to output this cycle E1122 03:55:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:55:16.411973 3798194 memory.go:184] no items to output this cycle I1122 03:55:16.412138 3798194 cpu.go:275] no items to output this cycle E1122 03:55:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:55:26.413266 3798194 memory.go:191] Add success. I1122 03:55:26.417550 3798194 cpu.go:282] Add success. I1122 03:55:26.442063 3798194 net.go:642] Add success. I1122 03:55:26.443723 3798194 net.go:778] primary dev: ETH0 I1122 03:55:26.443738 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:55:26.443753 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:55:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:55:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:55:29.450500 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:55:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:55:36.411980 3798194 memory.go:184] no items to output this cycle I1122 03:55:36.412133 3798194 cpu.go:275] no items to output this cycle E1122 03:55:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:55:46.411978 3798194 memory.go:184] no items to output this cycle I1122 03:55:46.412144 3798194 cpu.go:275] no items to output this cycle E1122 03:55:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:55:56.411951 3798194 memory.go:191] Add success. W1122 03:55:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:55:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:55:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:55:56.412347 3798194 cpu.go:282] Add success. I1122 03:55:56.441562 3798194 event_worker.go:152] Polling the log file for events... I1122 03:55:56.444682 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:55:56.445189 3798194 net.go:642] Add success. I1122 03:55:56.449245 3798194 net.go:778] primary dev: ETH0 I1122 03:55:56.449260 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:55:56.449272 3798194 net.go:659] Add eni metrics success, count: 7 E1122 03:55:57.450659 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 03:55:57.450673 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:55:57.450838 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:55:57.450894 3798194 disk_worker.go:723] disk space is not compliant W1122 03:55:57.450897 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:55:57.460026 3798194 disk_worker.go:509] system disk:vda1 I1122 03:55:57.460066 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 03:55:58.448279 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 03:55:58.448292 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:55:59.449801 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:55:59.449854 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:55:59.449873 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:55:59.449987 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 03:55:59.471423 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:56:01.821144 3798194 disk_info.go:125] begin check local disk info of client I1122 03:56:01.824993 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:56:01.825003 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000475800 0xc000475840] E1122 03:56:06.412092 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:56:06.412133 3798194 memory.go:184] no items to output this cycle I1122 03:56:06.412288 3798194 cpu.go:275] no items to output this cycle E1122 03:56:16.412414 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:56:16.412436 3798194 memory.go:184] no items to output this cycle I1122 03:56:16.412591 3798194 cpu.go:275] no items to output this cycle E1122 03:56:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:56:26.412268 3798194 memory.go:191] Add success. I1122 03:56:26.416625 3798194 cpu.go:282] Add success. I1122 03:56:26.442182 3798194 net.go:642] Add success. I1122 03:56:26.443832 3798194 net.go:778] primary dev: ETH0 I1122 03:56:26.443846 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:56:26.443860 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:56:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:56:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:56:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:56:36.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:56:36.413257 3798194 memory.go:184] no items to output this cycle I1122 03:56:36.413403 3798194 cpu.go:275] no items to output this cycle E1122 03:56:46.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:56:46.412263 3798194 memory.go:184] no items to output this cycle I1122 03:56:46.412411 3798194 cpu.go:275] no items to output this cycle E1122 03:56:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:56:56.411956 3798194 memory.go:191] Add success. W1122 03:56:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:56:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:56:56.412008 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:56:56.412360 3798194 cpu.go:282] Add success. I1122 03:56:56.441832 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:56:56.442489 3798194 net.go:642] Add success. I1122 03:56:56.447160 3798194 net.go:778] primary dev: ETH0 I1122 03:56:56.447176 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:56:56.447196 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:56:57.450864 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:56:57.451063 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:56:57.451074 3798194 disk_worker.go:723] disk space is not compliant W1122 03:56:57.451078 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:56:57.459869 3798194 disk_worker.go:509] system disk:vda1 I1122 03:56:57.459912 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:56:58.446400 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:56:59.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:56:59.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:56:59.450512 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:56:59.471021 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:57:01.826139 3798194 disk_info.go:125] begin check local disk info of client I1122 03:57:01.829983 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:57:01.829991 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b8440 0xc0003b8480] E1122 03:57:06.413159 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:57:06.413181 3798194 memory.go:184] no items to output this cycle I1122 03:57:06.413332 3798194 cpu.go:275] no items to output this cycle E1122 03:57:16.413372 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:57:16.413396 3798194 memory.go:184] no items to output this cycle I1122 03:57:16.413542 3798194 cpu.go:275] no items to output this cycle I1122 03:57:17.628387 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 03:57:17.628396 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 03:57:26.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:57:26.412268 3798194 memory.go:191] Add success. I1122 03:57:26.416820 3798194 cpu.go:282] Add success. I1122 03:57:26.441722 3798194 net.go:642] Add success. I1122 03:57:26.443316 3798194 net.go:778] primary dev: ETH0 I1122 03:57:26.443332 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:57:26.443350 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:57:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:57:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:57:29.450502 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:57:36.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:57:36.411986 3798194 memory.go:184] no items to output this cycle I1122 03:57:36.412139 3798194 cpu.go:275] no items to output this cycle E1122 03:57:46.412847 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:57:46.412869 3798194 memory.go:184] no items to output this cycle I1122 03:57:46.413013 3798194 cpu.go:275] no items to output this cycle E1122 03:57:56.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:57:56.412898 3798194 memory.go:191] Add success. W1122 03:57:56.412933 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:57:56.412944 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:57:56.412948 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:57:56.417466 3798194 cpu.go:282] Add success. I1122 03:57:56.441914 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:57:56.442530 3798194 net.go:642] Add success. I1122 03:57:56.444324 3798194 net.go:778] primary dev: ETH0 I1122 03:57:56.444339 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:57:56.444352 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:57:57.451052 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:57:57.451293 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:57:57.451305 3798194 disk_worker.go:723] disk space is not compliant W1122 03:57:57.451308 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:57:57.460085 3798194 disk_worker.go:509] system disk:vda1 I1122 03:57:57.460149 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:57:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:57:59.450105 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:57:59.450200 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:57:59.450227 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:57:59.470730 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:58:01.831144 3798194 disk_info.go:125] begin check local disk info of client I1122 03:58:01.835046 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:58:01.835054 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000456880 0xc0004568c0] E1122 03:58:06.413251 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:58:06.413277 3798194 memory.go:184] no items to output this cycle I1122 03:58:06.413424 3798194 cpu.go:275] no items to output this cycle E1122 03:58:16.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:58:16.412257 3798194 memory.go:184] no items to output this cycle I1122 03:58:16.412398 3798194 cpu.go:275] no items to output this cycle E1122 03:58:26.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:58:26.413263 3798194 memory.go:191] Add success. I1122 03:58:26.417671 3798194 cpu.go:282] Add success. I1122 03:58:26.442221 3798194 net.go:642] Add success. I1122 03:58:26.443770 3798194 net.go:778] primary dev: ETH0 I1122 03:58:26.443784 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:58:26.443795 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:58:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:58:29.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:58:29.450487 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:58:36.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:58:36.411991 3798194 memory.go:184] no items to output this cycle I1122 03:58:36.412146 3798194 cpu.go:275] no items to output this cycle E1122 03:58:46.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:58:46.411993 3798194 memory.go:184] no items to output this cycle I1122 03:58:46.412161 3798194 cpu.go:275] no items to output this cycle E1122 03:58:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:58:56.411953 3798194 memory.go:191] Add success. W1122 03:58:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:58:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:58:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:58:56.412376 3798194 cpu.go:282] Add success. I1122 03:58:56.441957 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:58:56.443218 3798194 net.go:642] Add success. I1122 03:58:56.451774 3798194 net.go:778] primary dev: ETH0 I1122 03:58:56.451793 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:58:56.451812 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:58:57.451139 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:58:57.451353 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:58:57.451365 3798194 disk_worker.go:723] disk space is not compliant W1122 03:58:57.451368 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:58:57.467725 3798194 disk_worker.go:509] system disk:vda1 I1122 03:58:57.467767 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:58:58.446001 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:58:59.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:58:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:58:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:58:59.471654 3798194 disk_local_worker.go:436] Get disk info: [] I1122 03:59:01.836143 3798194 disk_info.go:125] begin check local disk info of client I1122 03:59:01.840071 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 03:59:01.840081 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004e2280 0xc0004e22c0] E1122 03:59:06.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:59:06.412875 3798194 memory.go:184] no items to output this cycle I1122 03:59:06.413019 3798194 cpu.go:275] no items to output this cycle E1122 03:59:16.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:59:16.412262 3798194 memory.go:184] no items to output this cycle I1122 03:59:16.412409 3798194 cpu.go:275] no items to output this cycle E1122 03:59:26.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:59:26.412883 3798194 memory.go:191] Add success. I1122 03:59:26.417323 3798194 cpu.go:282] Add success. I1122 03:59:26.441989 3798194 net.go:642] Add success. I1122 03:59:26.443691 3798194 net.go:778] primary dev: ETH0 I1122 03:59:26.443706 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:59:26.443719 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:59:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:59:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:59:29.450494 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 03:59:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:59:36.411979 3798194 memory.go:184] no items to output this cycle I1122 03:59:36.412131 3798194 cpu.go:275] no items to output this cycle E1122 03:59:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:59:46.413265 3798194 memory.go:184] no items to output this cycle I1122 03:59:46.413411 3798194 cpu.go:275] no items to output this cycle E1122 03:59:56.411586 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 03:59:56.411619 3798194 memory.go:191] Add success. W1122 03:59:56.411651 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 03:59:56.411665 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 03:59:56.411668 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 03:59:56.415834 3798194 cpu.go:282] Add success. I1122 03:59:56.441302 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 03:59:56.442138 3798194 net.go:642] Add success. I1122 03:59:56.443608 3798194 net.go:778] primary dev: ETH0 I1122 03:59:56.443629 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 03:59:56.443643 3798194 net.go:659] Add eni metrics success, count: 7 I1122 03:59:57.450848 3798194 custom_config.go:106] query custom config with name: gpu W1122 03:59:57.451013 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 03:59:57.451067 3798194 disk_worker.go:723] disk space is not compliant W1122 03:59:57.451070 3798194 disk_worker.go:743] disk inode is not compliant I1122 03:59:57.460054 3798194 disk_worker.go:509] system disk:vda1 I1122 03:59:57.460096 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 03:59:58.445676 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 03:59:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 03:59:59.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 03:59:59.450501 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 03:59:59.470963 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:00:01.841146 3798194 disk_info.go:125] begin check local disk info of client I1122 04:00:01.845046 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:00:01.845055 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004169c0 0xc000416a00] E1122 04:00:06.413249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:00:06.413272 3798194 memory.go:184] no items to output this cycle I1122 04:00:06.413471 3798194 cpu.go:275] no items to output this cycle E1122 04:00:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:00:16.411975 3798194 memory.go:184] no items to output this cycle I1122 04:00:16.412139 3798194 cpu.go:275] no items to output this cycle I1122 04:00:17.628551 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:00:17.628561 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:00:26.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:00:26.412266 3798194 memory.go:191] Add success. I1122 04:00:26.417081 3798194 cpu.go:282] Add success. I1122 04:00:26.441606 3798194 net.go:642] Add success. I1122 04:00:26.443184 3798194 net.go:778] primary dev: ETH0 I1122 04:00:26.443200 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:00:26.443213 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:00:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:00:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:00:29.450498 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:00:36.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:00:36.412877 3798194 memory.go:184] no items to output this cycle I1122 04:00:36.413021 3798194 cpu.go:275] no items to output this cycle E1122 04:00:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:00:46.411983 3798194 memory.go:184] no items to output this cycle I1122 04:00:46.412142 3798194 cpu.go:275] no items to output this cycle E1122 04:00:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:00:56.411950 3798194 memory.go:191] Add success. W1122 04:00:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:00:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:00:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:00:56.412342 3798194 cpu.go:282] Add success. I1122 04:00:56.446984 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:00:56.447760 3798194 net.go:642] Add success. I1122 04:00:56.451676 3798194 net.go:778] primary dev: ETH0 I1122 04:00:56.451692 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:00:56.451707 3798194 net.go:659] Add eni metrics success, count: 7 E1122 04:00:57.450703 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 04:00:57.450718 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:00:57.450889 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:00:57.450945 3798194 disk_worker.go:723] disk space is not compliant W1122 04:00:57.450948 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:00:57.459408 3798194 disk_worker.go:509] system disk:vda1 I1122 04:00:57.459452 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 04:00:58.448036 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 04:00:58.448051 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 04:00:59.448736 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 04:00:59.449810 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:00:59.449860 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:00:59.449879 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:00:59.471418 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:01:01.846148 3798194 disk_info.go:125] begin check local disk info of client I1122 04:01:01.850002 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:01:01.850012 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c8a80 0xc0003c8ac0] E1122 04:01:06.412943 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:01:06.412966 3798194 memory.go:184] no items to output this cycle I1122 04:01:06.413138 3798194 cpu.go:275] no items to output this cycle E1122 04:01:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:01:16.411982 3798194 memory.go:184] no items to output this cycle I1122 04:01:16.412148 3798194 cpu.go:275] no items to output this cycle E1122 04:01:26.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:01:26.412271 3798194 memory.go:191] Add success. I1122 04:01:26.416677 3798194 cpu.go:282] Add success. I1122 04:01:26.441311 3798194 net.go:642] Add success. I1122 04:01:26.442733 3798194 net.go:778] primary dev: ETH0 I1122 04:01:26.442748 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:01:26.442761 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:01:29.450329 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:01:29.450413 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:01:29.450440 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:01:36.412474 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:01:36.412497 3798194 memory.go:184] no items to output this cycle I1122 04:01:36.412649 3798194 cpu.go:275] no items to output this cycle E1122 04:01:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:01:46.411986 3798194 memory.go:184] no items to output this cycle I1122 04:01:46.412144 3798194 cpu.go:275] no items to output this cycle E1122 04:01:56.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:01:56.412272 3798194 memory.go:191] Add success. W1122 04:01:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:01:56.412318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:01:56.412321 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:01:56.416592 3798194 cpu.go:282] Add success. I1122 04:01:56.440951 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:01:56.441727 3798194 net.go:642] Add success. I1122 04:01:56.443152 3798194 net.go:778] primary dev: ETH0 I1122 04:01:56.443167 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:01:56.443185 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:01:57.451324 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:01:57.451531 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:01:57.451542 3798194 disk_worker.go:723] disk space is not compliant W1122 04:01:57.451545 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:01:57.459970 3798194 disk_worker.go:509] system disk:vda1 I1122 04:01:57.460013 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:01:58.445687 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:01:59.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:01:59.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:01:59.450501 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:01:59.471028 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:02:01.850126 3798194 disk_info.go:125] begin check local disk info of client I1122 04:02:01.853951 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:02:01.853959 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000426a00 0xc000426a40] E1122 04:02:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:02:06.412257 3798194 memory.go:184] no items to output this cycle I1122 04:02:06.412404 3798194 cpu.go:275] no items to output this cycle E1122 04:02:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:02:16.413253 3798194 memory.go:184] no items to output this cycle I1122 04:02:16.413401 3798194 cpu.go:275] no items to output this cycle E1122 04:02:26.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:02:26.413275 3798194 memory.go:191] Add success. I1122 04:02:26.417409 3798194 cpu.go:282] Add success. I1122 04:02:26.441881 3798194 net.go:642] Add success. I1122 04:02:26.443395 3798194 net.go:778] primary dev: ETH0 I1122 04:02:26.443411 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:02:26.443424 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:02:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:02:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:02:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:02:36.411971 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:02:36.411994 3798194 memory.go:184] no items to output this cycle I1122 04:02:36.412158 3798194 cpu.go:275] no items to output this cycle E1122 04:02:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:02:46.411978 3798194 memory.go:184] no items to output this cycle I1122 04:02:46.412130 3798194 cpu.go:275] no items to output this cycle E1122 04:02:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:02:56.411958 3798194 memory.go:191] Add success. W1122 04:02:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:02:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:02:56.412008 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:02:56.412349 3798194 cpu.go:282] Add success. I1122 04:02:56.441791 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:02:56.442286 3798194 net.go:642] Add success. I1122 04:02:56.446707 3798194 net.go:778] primary dev: ETH0 I1122 04:02:56.446724 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:02:56.446744 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:02:57.450843 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:02:57.451057 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:02:57.451067 3798194 disk_worker.go:723] disk space is not compliant W1122 04:02:57.451070 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:02:57.459846 3798194 disk_worker.go:509] system disk:vda1 I1122 04:02:57.459889 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:02:58.446371 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:02:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:02:59.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:02:59.450505 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:02:59.471049 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:03:01.855146 3798194 disk_info.go:125] begin check local disk info of client I1122 04:03:01.858936 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:03:01.858946 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003be080 0xc0003be0c0] E1122 04:03:06.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:03:06.412261 3798194 memory.go:184] no items to output this cycle I1122 04:03:06.412404 3798194 cpu.go:275] no items to output this cycle E1122 04:03:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:03:16.411974 3798194 memory.go:184] no items to output this cycle I1122 04:03:16.412126 3798194 cpu.go:275] no items to output this cycle I1122 04:03:17.628715 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:03:17.628726 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:03:26.411545 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:03:26.411579 3798194 memory.go:191] Add success. I1122 04:03:26.416024 3798194 cpu.go:282] Add success. I1122 04:03:26.441557 3798194 net.go:642] Add success. I1122 04:03:26.443334 3798194 net.go:778] primary dev: ETH0 I1122 04:03:26.443348 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:03:26.443360 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:03:29.450657 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:03:29.450735 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:03:29.450760 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:03:36.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:03:36.412884 3798194 memory.go:184] no items to output this cycle I1122 04:03:36.413033 3798194 cpu.go:275] no items to output this cycle E1122 04:03:46.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:03:46.411990 3798194 memory.go:184] no items to output this cycle I1122 04:03:46.412149 3798194 cpu.go:275] no items to output this cycle E1122 04:03:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:03:56.411951 3798194 memory.go:191] Add success. W1122 04:03:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:03:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:03:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:03:56.412361 3798194 cpu.go:282] Add success. I1122 04:03:56.441854 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:03:56.442556 3798194 net.go:642] Add success. I1122 04:03:56.446612 3798194 net.go:778] primary dev: ETH0 I1122 04:03:56.446629 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:03:56.446647 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:03:57.451019 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:03:57.451268 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:03:57.451280 3798194 disk_worker.go:723] disk space is not compliant W1122 04:03:57.451283 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:03:57.463799 3798194 disk_worker.go:509] system disk:vda1 I1122 04:03:57.463840 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:03:58.445991 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:03:59.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:03:59.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:03:59.450515 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:03:59.471031 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:04:01.860735 3798194 disk_info.go:125] begin check local disk info of client I1122 04:04:01.864639 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:04:01.864648 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041e080 0xc00041e0c0] E1122 04:04:06.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:04:06.412257 3798194 memory.go:184] no items to output this cycle I1122 04:04:06.412402 3798194 cpu.go:275] no items to output this cycle E1122 04:04:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:04:16.411982 3798194 memory.go:184] no items to output this cycle I1122 04:04:16.412155 3798194 cpu.go:275] no items to output this cycle E1122 04:04:26.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:04:26.413282 3798194 memory.go:191] Add success. I1122 04:04:26.417811 3798194 cpu.go:282] Add success. I1122 04:04:26.441567 3798194 net.go:642] Add success. I1122 04:04:26.443093 3798194 net.go:778] primary dev: ETH0 I1122 04:04:26.443123 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:04:26.443139 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:04:29.450133 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:04:29.450216 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:04:29.450254 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:04:36.412345 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:04:36.412367 3798194 memory.go:184] no items to output this cycle I1122 04:04:36.412510 3798194 cpu.go:275] no items to output this cycle E1122 04:04:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:04:46.411981 3798194 memory.go:184] no items to output this cycle I1122 04:04:46.412135 3798194 cpu.go:275] no items to output this cycle E1122 04:04:56.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:04:56.411946 3798194 memory.go:191] Add success. W1122 04:04:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:04:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:04:56.411993 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:04:56.412339 3798194 cpu.go:282] Add success. I1122 04:04:56.441906 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:04:56.442506 3798194 net.go:642] Add success. I1122 04:04:56.446568 3798194 net.go:778] primary dev: ETH0 I1122 04:04:56.446582 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:04:56.446594 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:04:57.451086 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:04:57.451309 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:04:57.451320 3798194 disk_worker.go:723] disk space is not compliant W1122 04:04:57.451324 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:04:57.461536 3798194 disk_worker.go:509] system disk:vda1 I1122 04:04:57.461593 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:04:58.445692 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:04:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:04:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:04:59.450140 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:04:59.470672 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:05:01.864741 3798194 disk_info.go:125] begin check local disk info of client I1122 04:05:01.868599 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:05:01.868608 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00045c880 0xc00045c8c0] E1122 04:05:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:05:06.411981 3798194 memory.go:184] no items to output this cycle I1122 04:05:06.412166 3798194 cpu.go:275] no items to output this cycle E1122 04:05:16.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:05:16.412272 3798194 memory.go:184] no items to output this cycle I1122 04:05:16.412433 3798194 cpu.go:275] no items to output this cycle E1122 04:05:26.412770 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:05:26.412810 3798194 memory.go:191] Add success. I1122 04:05:26.417758 3798194 cpu.go:282] Add success. I1122 04:05:26.441467 3798194 net.go:642] Add success. I1122 04:05:26.443014 3798194 net.go:778] primary dev: ETH0 I1122 04:05:26.443030 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:05:26.443043 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:05:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:05:29.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:05:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:05:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:05:36.411980 3798194 memory.go:184] no items to output this cycle I1122 04:05:36.412139 3798194 cpu.go:275] no items to output this cycle E1122 04:05:46.413066 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:05:46.413087 3798194 memory.go:184] no items to output this cycle I1122 04:05:46.413257 3798194 cpu.go:275] no items to output this cycle E1122 04:05:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:05:56.411958 3798194 memory.go:191] Add success. W1122 04:05:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:05:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:05:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:05:56.412353 3798194 cpu.go:282] Add success. I1122 04:05:56.442213 3798194 event_worker.go:152] Polling the log file for events... I1122 04:05:56.445311 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:05:56.445847 3798194 net.go:642] Add success. I1122 04:05:56.450806 3798194 net.go:778] primary dev: ETH0 I1122 04:05:56.450831 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:05:56.450843 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:05:56.450860 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795289756, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 04:05:56.453670 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 04:05:57.446437 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 04:05:57.446452 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:05:57.446693 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:05:57.446705 3798194 disk_worker.go:723] disk space is not compliant W1122 04:05:57.446708 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:05:57.449523 3798194 disk_worker.go:509] system disk:vda1 I1122 04:05:57.449562 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 04:05:58.448152 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 04:05:58.448168 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 04:05:59.449041 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 04:05:59.450121 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:05:59.450178 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:05:59.450196 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:05:59.471714 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:06:01.869146 3798194 disk_info.go:125] begin check local disk info of client I1122 04:06:01.873020 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:06:01.873029 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000323880 0xc0003238c0] E1122 04:06:06.411980 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:06:06.412002 3798194 memory.go:184] no items to output this cycle I1122 04:06:06.412165 3798194 cpu.go:275] no items to output this cycle E1122 04:06:16.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:06:16.411991 3798194 memory.go:184] no items to output this cycle I1122 04:06:16.412144 3798194 cpu.go:275] no items to output this cycle I1122 04:06:17.628887 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:06:17.628895 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:06:26.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:06:26.412280 3798194 memory.go:191] Add success. I1122 04:06:26.416468 3798194 cpu.go:282] Add success. I1122 04:06:26.442138 3798194 net.go:642] Add success. I1122 04:06:26.443685 3798194 net.go:778] primary dev: ETH0 I1122 04:06:26.443705 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:06:26.443719 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:06:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:06:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:06:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:06:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:06:36.412258 3798194 memory.go:184] no items to output this cycle I1122 04:06:36.412402 3798194 cpu.go:275] no items to output this cycle E1122 04:06:46.411972 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:06:46.411994 3798194 memory.go:184] no items to output this cycle I1122 04:06:46.412358 3798194 cpu.go:275] no items to output this cycle E1122 04:06:56.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:06:56.411945 3798194 memory.go:191] Add success. W1122 04:06:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:06:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:06:56.411993 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:06:56.412371 3798194 cpu.go:282] Add success. I1122 04:06:56.441947 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:06:56.442547 3798194 net.go:642] Add success. I1122 04:06:56.446751 3798194 net.go:778] primary dev: ETH0 I1122 04:06:56.446767 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:06:56.446780 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:06:57.451167 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:06:57.451385 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:06:57.451396 3798194 disk_worker.go:723] disk space is not compliant W1122 04:06:57.451399 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:06:57.459921 3798194 disk_worker.go:509] system disk:vda1 I1122 04:06:57.459962 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:06:58.446515 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:06:59.450620 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:06:59.450698 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:06:59.450725 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:06:59.471190 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:07:01.874148 3798194 disk_info.go:125] begin check local disk info of client I1122 04:07:01.878031 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:07:01.878041 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000383240 0xc000383280] E1122 04:07:06.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:07:06.413250 3798194 memory.go:184] no items to output this cycle I1122 04:07:06.413397 3798194 cpu.go:275] no items to output this cycle E1122 04:07:16.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:07:16.413268 3798194 memory.go:184] no items to output this cycle I1122 04:07:16.413413 3798194 cpu.go:275] no items to output this cycle E1122 04:07:26.411992 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:07:26.412025 3798194 memory.go:191] Add success. I1122 04:07:26.416170 3798194 cpu.go:282] Add success. I1122 04:07:26.441760 3798194 net.go:642] Add success. I1122 04:07:26.443198 3798194 net.go:778] primary dev: ETH0 I1122 04:07:26.443212 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:07:26.443225 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:07:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:07:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:07:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:07:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:07:36.413262 3798194 memory.go:184] no items to output this cycle I1122 04:07:36.413407 3798194 cpu.go:275] no items to output this cycle E1122 04:07:46.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:07:46.411972 3798194 memory.go:184] no items to output this cycle I1122 04:07:46.412127 3798194 cpu.go:275] no items to output this cycle E1122 04:07:56.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:07:56.413269 3798194 memory.go:191] Add success. W1122 04:07:56.413301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:07:56.413313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:07:56.413316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:07:56.417787 3798194 cpu.go:282] Add success. I1122 04:07:56.441219 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:07:56.441785 3798194 net.go:642] Add success. I1122 04:07:56.443292 3798194 net.go:778] primary dev: ETH0 I1122 04:07:56.443306 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:07:56.443319 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:07:57.452240 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:07:57.452449 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:07:57.452459 3798194 disk_worker.go:723] disk space is not compliant W1122 04:07:57.452463 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:07:57.464377 3798194 disk_worker.go:509] system disk:vda1 I1122 04:07:57.464426 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:07:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:07:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:07:59.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:07:59.450512 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:07:59.471051 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:08:01.879147 3798194 disk_info.go:125] begin check local disk info of client I1122 04:08:01.882989 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:08:01.882998 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004807c0 0xc000480800] E1122 04:08:06.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:08:06.413252 3798194 memory.go:184] no items to output this cycle I1122 04:08:06.413395 3798194 cpu.go:275] no items to output this cycle E1122 04:08:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:08:16.411983 3798194 memory.go:184] no items to output this cycle I1122 04:08:16.412140 3798194 cpu.go:275] no items to output this cycle E1122 04:08:26.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:08:26.412899 3798194 memory.go:191] Add success. I1122 04:08:26.417256 3798194 cpu.go:282] Add success. I1122 04:08:26.441751 3798194 net.go:642] Add success. I1122 04:08:26.443424 3798194 net.go:778] primary dev: ETH0 I1122 04:08:26.443439 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:08:26.443454 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:08:29.450200 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:08:29.450280 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:08:29.450306 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:08:36.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:08:36.411986 3798194 memory.go:184] no items to output this cycle I1122 04:08:36.412141 3798194 cpu.go:275] no items to output this cycle E1122 04:08:46.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:08:46.413269 3798194 memory.go:184] no items to output this cycle I1122 04:08:46.413418 3798194 cpu.go:275] no items to output this cycle E1122 04:08:56.411910 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:08:56.411942 3798194 memory.go:191] Add success. W1122 04:08:56.411969 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:08:56.411981 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:08:56.411984 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:08:56.412339 3798194 cpu.go:282] Add success. I1122 04:08:56.441799 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:08:56.442300 3798194 net.go:642] Add success. I1122 04:08:56.446912 3798194 net.go:778] primary dev: ETH0 I1122 04:08:56.446929 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:08:56.446941 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:08:57.451065 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:08:57.451253 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:08:57.451307 3798194 disk_worker.go:723] disk space is not compliant W1122 04:08:57.451310 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:08:57.460057 3798194 disk_worker.go:509] system disk:vda1 I1122 04:08:57.460099 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:08:58.445973 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:08:59.450074 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:08:59.450166 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:08:59.450195 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:08:59.470689 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:09:01.884138 3798194 disk_info.go:125] begin check local disk info of client I1122 04:09:01.888054 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:09:01.888063 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a57c0 0xc0003a5800] E1122 04:09:06.413210 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:09:06.413232 3798194 memory.go:184] no items to output this cycle I1122 04:09:06.413378 3798194 cpu.go:275] no items to output this cycle E1122 04:09:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:09:16.411979 3798194 memory.go:184] no items to output this cycle I1122 04:09:16.412138 3798194 cpu.go:275] no items to output this cycle I1122 04:09:17.629050 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:09:17.629058 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:09:26.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:09:26.412264 3798194 memory.go:191] Add success. I1122 04:09:26.416623 3798194 cpu.go:282] Add success. I1122 04:09:26.441493 3798194 net.go:642] Add success. I1122 04:09:26.443001 3798194 net.go:778] primary dev: ETH0 I1122 04:09:26.443016 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:09:26.443030 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:09:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:09:29.450133 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:09:29.450169 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:09:36.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:09:36.411977 3798194 memory.go:184] no items to output this cycle I1122 04:09:36.412129 3798194 cpu.go:275] no items to output this cycle E1122 04:09:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:09:46.411982 3798194 memory.go:184] no items to output this cycle I1122 04:09:46.412140 3798194 cpu.go:275] no items to output this cycle E1122 04:09:56.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:09:56.412883 3798194 memory.go:191] Add success. W1122 04:09:56.412911 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:09:56.412922 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:09:56.412926 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:09:56.417041 3798194 cpu.go:282] Add success. I1122 04:09:56.441426 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:09:56.441953 3798194 net.go:642] Add success. I1122 04:09:56.443445 3798194 net.go:778] primary dev: ETH0 I1122 04:09:56.443460 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:09:56.443472 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:09:57.450635 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:09:57.450792 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:09:57.450844 3798194 disk_worker.go:723] disk space is not compliant W1122 04:09:57.450847 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:09:57.459297 3798194 disk_worker.go:509] system disk:vda1 I1122 04:09:57.459339 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:09:58.445994 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:09:59.450089 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:09:59.450179 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:09:59.450205 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:09:59.470672 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:10:01.889146 3798194 disk_info.go:125] begin check local disk info of client I1122 04:10:01.893025 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:10:01.893035 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a5100 0xc0003a5140] E1122 04:10:06.413106 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:10:06.413151 3798194 memory.go:184] no items to output this cycle I1122 04:10:06.413318 3798194 cpu.go:275] no items to output this cycle E1122 04:10:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:10:16.413257 3798194 memory.go:184] no items to output this cycle I1122 04:10:16.413404 3798194 cpu.go:275] no items to output this cycle E1122 04:10:26.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:10:26.412274 3798194 memory.go:191] Add success. I1122 04:10:26.416509 3798194 cpu.go:282] Add success. I1122 04:10:26.442182 3798194 net.go:642] Add success. I1122 04:10:26.443581 3798194 net.go:778] primary dev: ETH0 I1122 04:10:26.443595 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:10:26.443607 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:10:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:10:29.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:10:29.450145 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:10:36.412868 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:10:36.412889 3798194 memory.go:184] no items to output this cycle I1122 04:10:36.413027 3798194 cpu.go:275] no items to output this cycle E1122 04:10:46.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:10:46.413253 3798194 memory.go:184] no items to output this cycle I1122 04:10:46.413395 3798194 cpu.go:275] no items to output this cycle E1122 04:10:56.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:10:56.412259 3798194 memory.go:191] Add success. W1122 04:10:56.412291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:10:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:10:56.412305 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:10:56.416554 3798194 cpu.go:282] Add success. I1122 04:10:56.443972 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:10:56.444482 3798194 net.go:642] Add success. I1122 04:10:56.445832 3798194 net.go:778] primary dev: ETH0 I1122 04:10:56.445847 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:10:56.445859 3798194 net.go:659] Add eni metrics success, count: 7 E1122 04:10:57.451100 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 04:10:57.451131 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:10:57.451337 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:10:57.451357 3798194 disk_worker.go:723] disk space is not compliant W1122 04:10:57.451361 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:10:57.460062 3798194 disk_worker.go:509] system disk:vda1 I1122 04:10:57.460103 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 04:10:58.448436 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 04:10:58.448450 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 04:10:59.448970 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 04:10:59.450046 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:10:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:10:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:10:59.471657 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:11:01.894143 3798194 disk_info.go:125] begin check local disk info of client I1122 04:11:01.897973 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:11:01.897983 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004da180 0xc0004da1c0] E1122 04:11:06.412875 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:11:06.412897 3798194 memory.go:184] no items to output this cycle I1122 04:11:06.413038 3798194 cpu.go:275] no items to output this cycle E1122 04:11:16.412843 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:11:16.412865 3798194 memory.go:184] no items to output this cycle I1122 04:11:16.413010 3798194 cpu.go:275] no items to output this cycle E1122 04:11:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:11:26.411955 3798194 memory.go:191] Add success. I1122 04:11:26.412321 3798194 cpu.go:282] Add success. I1122 04:11:26.441943 3798194 net.go:642] Add success. I1122 04:11:26.446101 3798194 net.go:778] primary dev: ETH0 I1122 04:11:26.446133 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:11:26.446146 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:11:29.450128 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:11:29.450237 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:11:29.450272 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:11:36.412770 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:11:36.412796 3798194 memory.go:184] no items to output this cycle I1122 04:11:36.412945 3798194 cpu.go:275] no items to output this cycle E1122 04:11:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:11:46.411974 3798194 memory.go:184] no items to output this cycle I1122 04:11:46.412143 3798194 cpu.go:275] no items to output this cycle E1122 04:11:56.412738 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:11:56.412770 3798194 memory.go:191] Add success. W1122 04:11:56.412801 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:11:56.412814 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:11:56.412818 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:11:56.418394 3798194 cpu.go:282] Add success. I1122 04:11:56.441166 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:11:56.441639 3798194 net.go:642] Add success. I1122 04:11:56.443078 3798194 net.go:778] primary dev: ETH0 I1122 04:11:56.443094 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:11:56.443106 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:11:57.449174 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:11:57.449368 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:11:57.449427 3798194 disk_worker.go:723] disk space is not compliant W1122 04:11:57.449431 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:11:57.451216 3798194 disk_worker.go:509] system disk:vda1 I1122 04:11:57.451256 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:11:58.445985 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:11:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:11:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:11:59.450148 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:11:59.471647 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:12:01.899362 3798194 disk_info.go:125] begin check local disk info of client I1122 04:12:01.903326 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:12:01.903335 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fd340 0xc0003fd380] E1122 04:12:06.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:12:06.411973 3798194 memory.go:184] no items to output this cycle I1122 04:12:06.412124 3798194 cpu.go:275] no items to output this cycle E1122 04:12:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:12:16.413256 3798194 memory.go:184] no items to output this cycle I1122 04:12:16.413400 3798194 cpu.go:275] no items to output this cycle I1122 04:12:17.629232 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:12:17.629240 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:12:26.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:12:26.412899 3798194 memory.go:191] Add success. I1122 04:12:26.416798 3798194 cpu.go:282] Add success. I1122 04:12:26.441588 3798194 net.go:642] Add success. I1122 04:12:26.443083 3798194 net.go:778] primary dev: ETH0 I1122 04:12:26.443098 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:12:26.443121 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:12:29.450386 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:12:29.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:12:29.450492 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:12:36.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:12:36.411984 3798194 memory.go:184] no items to output this cycle I1122 04:12:36.412143 3798194 cpu.go:275] no items to output this cycle E1122 04:12:46.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:12:46.412888 3798194 memory.go:184] no items to output this cycle I1122 04:12:46.413039 3798194 cpu.go:275] no items to output this cycle E1122 04:12:56.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:12:56.412259 3798194 memory.go:191] Add success. W1122 04:12:56.412301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:12:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:12:56.412316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:12:56.416799 3798194 cpu.go:282] Add success. I1122 04:12:56.441237 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:12:56.441749 3798194 net.go:642] Add success. I1122 04:12:56.443200 3798194 net.go:778] primary dev: ETH0 I1122 04:12:56.443215 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:12:56.443227 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:12:57.450986 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:12:57.451171 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:12:57.451237 3798194 disk_worker.go:723] disk space is not compliant W1122 04:12:57.451241 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:12:57.459963 3798194 disk_worker.go:509] system disk:vda1 I1122 04:12:57.460003 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:12:58.446374 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:12:59.450128 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:12:59.450207 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:12:59.450232 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:12:59.470693 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:13:01.905147 3798194 disk_info.go:125] begin check local disk info of client I1122 04:13:01.909026 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:13:01.909036 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000326c40 0xc000326c80] E1122 04:13:06.412868 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:13:06.412894 3798194 memory.go:184] no items to output this cycle I1122 04:13:06.413034 3798194 cpu.go:275] no items to output this cycle E1122 04:13:16.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:13:16.413251 3798194 memory.go:184] no items to output this cycle I1122 04:13:16.413398 3798194 cpu.go:275] no items to output this cycle E1122 04:13:26.411453 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:13:26.411489 3798194 memory.go:191] Add success. I1122 04:13:26.415393 3798194 cpu.go:282] Add success. I1122 04:13:26.442046 3798194 net.go:642] Add success. I1122 04:13:26.443585 3798194 net.go:778] primary dev: ETH0 I1122 04:13:26.443600 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:13:26.443615 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:13:29.449675 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:13:29.449748 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:13:29.449773 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:13:36.412870 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:13:36.412894 3798194 memory.go:184] no items to output this cycle I1122 04:13:36.413046 3798194 cpu.go:275] no items to output this cycle E1122 04:13:46.412391 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:13:46.412420 3798194 memory.go:184] no items to output this cycle I1122 04:13:46.412641 3798194 cpu.go:275] no items to output this cycle E1122 04:13:56.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:13:56.413257 3798194 memory.go:191] Add success. W1122 04:13:56.413285 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:13:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:13:56.413300 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:13:56.417727 3798194 cpu.go:282] Add success. I1122 04:13:56.441874 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:13:56.442425 3798194 net.go:642] Add success. I1122 04:13:56.444043 3798194 net.go:778] primary dev: ETH0 I1122 04:13:56.444058 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:13:56.444072 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:13:57.445619 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:13:57.445841 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:13:57.445908 3798194 disk_worker.go:723] disk space is not compliant W1122 04:13:57.445912 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:13:57.450990 3798194 disk_worker.go:509] system disk:vda1 I1122 04:13:57.451031 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:13:58.445989 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:13:59.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:13:59.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:13:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:13:59.471644 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:14:01.911143 3798194 disk_info.go:125] begin check local disk info of client I1122 04:14:01.915043 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:14:01.915053 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e480 0xc00030e4c0] E1122 04:14:06.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:14:06.413265 3798194 memory.go:184] no items to output this cycle I1122 04:14:06.413405 3798194 cpu.go:275] no items to output this cycle E1122 04:14:16.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:14:16.411991 3798194 memory.go:184] no items to output this cycle I1122 04:14:16.412159 3798194 cpu.go:275] no items to output this cycle E1122 04:14:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:14:26.411962 3798194 memory.go:191] Add success. I1122 04:14:26.412296 3798194 cpu.go:282] Add success. I1122 04:14:26.441888 3798194 net.go:642] Add success. I1122 04:14:26.446474 3798194 net.go:778] primary dev: ETH0 I1122 04:14:26.446491 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:14:26.446505 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:14:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:14:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:14:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:14:36.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:14:36.413248 3798194 memory.go:184] no items to output this cycle I1122 04:14:36.413403 3798194 cpu.go:275] no items to output this cycle E1122 04:14:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:14:46.413255 3798194 memory.go:184] no items to output this cycle I1122 04:14:46.413402 3798194 cpu.go:275] no items to output this cycle E1122 04:14:56.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:14:56.413275 3798194 memory.go:191] Add success. W1122 04:14:56.413315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:14:56.413326 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:14:56.413329 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:14:56.417469 3798194 cpu.go:282] Add success. I1122 04:14:56.441828 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:14:56.442424 3798194 net.go:642] Add success. I1122 04:14:56.444042 3798194 net.go:778] primary dev: ETH0 I1122 04:14:56.444058 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:14:56.444072 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:14:57.450694 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:14:57.450900 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:14:57.450911 3798194 disk_worker.go:723] disk space is not compliant W1122 04:14:57.450914 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:14:57.459214 3798194 disk_worker.go:509] system disk:vda1 I1122 04:14:57.459255 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:14:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:14:59.450100 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:14:59.450183 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:14:59.450210 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:14:59.470670 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:15:01.915149 3798194 disk_info.go:125] begin check local disk info of client I1122 04:15:01.918958 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:15:01.918967 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000417bc0 0xc000417c00] E1122 04:15:06.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:15:06.411996 3798194 memory.go:184] no items to output this cycle I1122 04:15:06.412165 3798194 cpu.go:275] no items to output this cycle E1122 04:15:16.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:15:16.411996 3798194 memory.go:184] no items to output this cycle I1122 04:15:16.412157 3798194 cpu.go:275] no items to output this cycle I1122 04:15:17.629742 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:15:17.629751 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:15:26.413638 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:15:26.413670 3798194 memory.go:191] Add success. I1122 04:15:26.417904 3798194 cpu.go:282] Add success. I1122 04:15:26.441573 3798194 net.go:642] Add success. I1122 04:15:26.443259 3798194 net.go:778] primary dev: ETH0 I1122 04:15:26.443273 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:15:26.443286 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:15:29.450067 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:15:29.450148 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:15:29.450188 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:15:36.412393 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:15:36.412415 3798194 memory.go:184] no items to output this cycle I1122 04:15:36.412562 3798194 cpu.go:275] no items to output this cycle E1122 04:15:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:15:46.413260 3798194 memory.go:184] no items to output this cycle I1122 04:15:46.413405 3798194 cpu.go:275] no items to output this cycle E1122 04:15:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:15:56.411944 3798194 memory.go:191] Add success. W1122 04:15:56.411972 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:15:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:15:56.411989 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:15:56.412344 3798194 cpu.go:282] Add success. I1122 04:15:56.441661 3798194 event_worker.go:152] Polling the log file for events... I1122 04:15:56.444845 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:15:56.445431 3798194 net.go:642] Add success. I1122 04:15:56.450249 3798194 net.go:778] primary dev: ETH0 I1122 04:15:56.450264 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:15:56.450279 3798194 net.go:659] Add eni metrics success, count: 7 E1122 04:15:57.445579 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 04:15:57.445604 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:15:57.445796 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:15:57.445857 3798194 disk_worker.go:723] disk space is not compliant W1122 04:15:57.445860 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:15:57.450795 3798194 disk_worker.go:509] system disk:vda1 I1122 04:15:57.450837 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 04:15:58.448131 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 04:15:58.448147 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 04:15:59.448636 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 04:15:59.449712 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:15:59.449763 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:15:59.449787 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:15:59.471329 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:16:01.921497 3798194 disk_info.go:125] begin check local disk info of client I1122 04:16:01.925801 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:16:01.925810 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037e4c0 0xc00037e500] E1122 04:16:06.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:16:06.411988 3798194 memory.go:184] no items to output this cycle I1122 04:16:06.412149 3798194 cpu.go:275] no items to output this cycle E1122 04:16:16.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:16:16.413269 3798194 memory.go:184] no items to output this cycle I1122 04:16:16.413425 3798194 cpu.go:275] no items to output this cycle E1122 04:16:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:16:26.412269 3798194 memory.go:191] Add success. I1122 04:16:26.416180 3798194 cpu.go:282] Add success. I1122 04:16:26.441752 3798194 net.go:642] Add success. I1122 04:16:26.443106 3798194 net.go:778] primary dev: ETH0 I1122 04:16:26.443134 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:16:26.443147 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:16:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:16:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:16:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:16:36.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:16:36.411976 3798194 memory.go:184] no items to output this cycle I1122 04:16:36.412130 3798194 cpu.go:275] no items to output this cycle E1122 04:16:46.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:16:46.412249 3798194 memory.go:184] no items to output this cycle I1122 04:16:46.412588 3798194 cpu.go:275] no items to output this cycle E1122 04:16:56.411930 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:16:56.411962 3798194 memory.go:191] Add success. W1122 04:16:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:16:56.412009 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:16:56.412012 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:16:56.412389 3798194 cpu.go:282] Add success. I1122 04:16:56.440906 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:16:56.441480 3798194 net.go:642] Add success. I1122 04:16:56.445832 3798194 net.go:778] primary dev: ETH0 I1122 04:16:56.445846 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:16:56.445862 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:16:57.445585 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:16:57.445794 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:16:57.445859 3798194 disk_worker.go:723] disk space is not compliant W1122 04:16:57.445863 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:16:57.450810 3798194 disk_worker.go:509] system disk:vda1 I1122 04:16:57.450849 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:16:58.445986 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:16:59.450171 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:16:59.450260 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:16:59.450285 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:16:59.470733 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:17:01.925915 3798194 disk_info.go:125] begin check local disk info of client I1122 04:17:01.929897 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:17:01.929907 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00042c600 0xc00042c640] E1122 04:17:06.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:17:06.411988 3798194 memory.go:184] no items to output this cycle I1122 04:17:06.412155 3798194 cpu.go:275] no items to output this cycle E1122 04:17:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:17:16.411981 3798194 memory.go:184] no items to output this cycle I1122 04:17:16.412140 3798194 cpu.go:275] no items to output this cycle E1122 04:17:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:17:26.411952 3798194 memory.go:191] Add success. I1122 04:17:26.412316 3798194 cpu.go:282] Add success. I1122 04:17:26.441895 3798194 net.go:642] Add success. I1122 04:17:26.446265 3798194 net.go:778] primary dev: ETH0 I1122 04:17:26.446281 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:17:26.446293 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:17:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:17:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:17:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:17:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:17:36.412257 3798194 memory.go:184] no items to output this cycle I1122 04:17:36.412401 3798194 cpu.go:275] no items to output this cycle E1122 04:17:46.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:17:46.412256 3798194 memory.go:184] no items to output this cycle I1122 04:17:46.412399 3798194 cpu.go:275] no items to output this cycle E1122 04:17:56.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:17:56.412273 3798194 memory.go:191] Add success. W1122 04:17:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:17:56.412319 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:17:56.412322 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:17:56.416783 3798194 cpu.go:282] Add success. I1122 04:17:56.441619 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:17:56.442179 3798194 net.go:642] Add success. I1122 04:17:56.443686 3798194 net.go:778] primary dev: ETH0 I1122 04:17:56.443701 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:17:56.443716 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:17:57.450949 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:17:57.451173 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:17:57.451184 3798194 disk_worker.go:723] disk space is not compliant W1122 04:17:57.451187 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:17:57.460353 3798194 disk_worker.go:509] system disk:vda1 I1122 04:17:57.460405 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:17:58.446372 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:17:59.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:17:59.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:17:59.450499 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:17:59.471020 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:18:01.931147 3798194 disk_info.go:125] begin check local disk info of client I1122 04:18:01.934962 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:18:01.934970 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00045d500 0xc00045d540] E1122 04:18:06.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:18:06.413268 3798194 memory.go:184] no items to output this cycle I1122 04:18:06.413411 3798194 cpu.go:275] no items to output this cycle E1122 04:18:16.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:18:16.413270 3798194 memory.go:184] no items to output this cycle I1122 04:18:16.413414 3798194 cpu.go:275] no items to output this cycle I1122 04:18:17.630209 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:18:17.630218 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:18:26.412504 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:18:26.412543 3798194 memory.go:191] Add success. I1122 04:18:26.416856 3798194 cpu.go:282] Add success. I1122 04:18:26.441557 3798194 net.go:642] Add success. I1122 04:18:26.443004 3798194 net.go:778] primary dev: ETH0 I1122 04:18:26.443020 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:18:26.443032 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:18:29.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:18:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:18:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:18:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:18:36.411977 3798194 memory.go:184] no items to output this cycle I1122 04:18:36.412131 3798194 cpu.go:275] no items to output this cycle E1122 04:18:46.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:18:46.412261 3798194 memory.go:184] no items to output this cycle I1122 04:18:46.412408 3798194 cpu.go:275] no items to output this cycle E1122 04:18:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:18:56.411949 3798194 memory.go:191] Add success. W1122 04:18:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:18:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:18:56.411993 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:18:56.412336 3798194 cpu.go:282] Add success. I1122 04:18:56.441920 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:18:56.442551 3798194 net.go:642] Add success. I1122 04:18:56.446352 3798194 net.go:778] primary dev: ETH0 I1122 04:18:56.446367 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:18:56.446381 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:18:57.450696 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:18:57.450892 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:18:57.450904 3798194 disk_worker.go:723] disk space is not compliant W1122 04:18:57.450907 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:18:57.459196 3798194 disk_worker.go:509] system disk:vda1 I1122 04:18:57.459240 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:18:58.445696 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:18:59.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:18:59.450086 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:18:59.450121 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:18:59.471631 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:19:01.936136 3798194 disk_info.go:125] begin check local disk info of client I1122 04:19:01.939987 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:19:01.939995 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003dac40 0xc0003dac80] E1122 04:19:06.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:19:06.411980 3798194 memory.go:184] no items to output this cycle I1122 04:19:06.412134 3798194 cpu.go:275] no items to output this cycle E1122 04:19:16.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:19:16.411994 3798194 memory.go:184] no items to output this cycle I1122 04:19:16.412151 3798194 cpu.go:275] no items to output this cycle E1122 04:19:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:19:26.411966 3798194 memory.go:191] Add success. I1122 04:19:26.412352 3798194 cpu.go:282] Add success. I1122 04:19:26.441951 3798194 net.go:642] Add success. I1122 04:19:26.445967 3798194 net.go:778] primary dev: ETH0 I1122 04:19:26.445981 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:19:26.445995 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:19:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:19:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:19:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:19:36.412980 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:19:36.413002 3798194 memory.go:184] no items to output this cycle I1122 04:19:36.413168 3798194 cpu.go:275] no items to output this cycle E1122 04:19:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:19:46.413252 3798194 memory.go:184] no items to output this cycle I1122 04:19:46.413397 3798194 cpu.go:275] no items to output this cycle E1122 04:19:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:19:56.411946 3798194 memory.go:191] Add success. W1122 04:19:56.411981 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:19:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:19:56.411996 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:19:56.412353 3798194 cpu.go:282] Add success. I1122 04:19:56.441832 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:19:56.442435 3798194 net.go:642] Add success. I1122 04:19:56.446339 3798194 net.go:778] primary dev: ETH0 I1122 04:19:56.446354 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:19:56.446367 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:19:57.453786 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:19:57.453952 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:19:57.454007 3798194 disk_worker.go:723] disk space is not compliant W1122 04:19:57.454010 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:19:57.462367 3798194 disk_worker.go:509] system disk:vda1 I1122 04:19:57.462408 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:19:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:19:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:19:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:19:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:19:59.471681 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:20:01.942147 3798194 disk_info.go:125] begin check local disk info of client I1122 04:20:01.945953 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:20:01.945962 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002eb400 0xc0002eb440] E1122 04:20:06.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:20:06.413267 3798194 memory.go:184] no items to output this cycle I1122 04:20:06.413409 3798194 cpu.go:275] no items to output this cycle E1122 04:20:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:20:16.411982 3798194 memory.go:184] no items to output this cycle I1122 04:20:16.412185 3798194 cpu.go:275] no items to output this cycle E1122 04:20:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:20:26.413266 3798194 memory.go:191] Add success. I1122 04:20:26.418081 3798194 cpu.go:282] Add success. I1122 04:20:26.441893 3798194 net.go:642] Add success. I1122 04:20:26.443342 3798194 net.go:778] primary dev: ETH0 I1122 04:20:26.443358 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:20:26.443371 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:20:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:20:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:20:29.450498 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:20:36.413259 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:20:36.413286 3798194 memory.go:184] no items to output this cycle I1122 04:20:36.413429 3798194 cpu.go:275] no items to output this cycle E1122 04:20:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:20:46.413269 3798194 memory.go:184] no items to output this cycle I1122 04:20:46.413414 3798194 cpu.go:275] no items to output this cycle E1122 04:20:56.411613 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:20:56.411647 3798194 memory.go:191] Add success. W1122 04:20:56.411682 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:20:56.411693 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:20:56.411697 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:20:56.415885 3798194 cpu.go:282] Add success. I1122 04:20:56.445254 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:20:56.445723 3798194 net.go:642] Add success. I1122 04:20:56.452708 3798194 net.go:778] primary dev: ETH0 I1122 04:20:56.452723 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:20:56.452735 3798194 net.go:659] Add eni metrics success, count: 7 E1122 04:20:57.451031 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 04:20:57.451045 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:20:57.451277 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:20:57.451287 3798194 disk_worker.go:723] disk space is not compliant W1122 04:20:57.451290 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:20:57.460866 3798194 disk_worker.go:509] system disk:vda1 I1122 04:20:57.460919 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 04:20:58.448073 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 04:20:58.448087 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 04:20:59.448739 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 04:20:59.449825 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:20:59.449880 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:20:59.449900 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:20:59.471455 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:21:01.947149 3798194 disk_info.go:125] begin check local disk info of client I1122 04:21:01.950957 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:21:01.950965 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000262f00 0xc000262f40] E1122 04:21:06.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:21:06.413268 3798194 memory.go:184] no items to output this cycle I1122 04:21:06.413420 3798194 cpu.go:275] no items to output this cycle E1122 04:21:16.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:21:16.411967 3798194 memory.go:184] no items to output this cycle I1122 04:21:16.412120 3798194 cpu.go:275] no items to output this cycle I1122 04:21:17.630387 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:21:17.630395 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:21:26.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:21:26.412276 3798194 memory.go:191] Add success. I1122 04:21:26.416553 3798194 cpu.go:282] Add success. I1122 04:21:26.442251 3798194 net.go:642] Add success. I1122 04:21:26.443845 3798194 net.go:778] primary dev: ETH0 I1122 04:21:26.443859 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:21:26.443873 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:21:29.450320 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:21:29.450402 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:21:29.450429 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:21:36.413540 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:21:36.413565 3798194 memory.go:184] no items to output this cycle I1122 04:21:36.413711 3798194 cpu.go:275] no items to output this cycle E1122 04:21:46.413353 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:21:46.413375 3798194 memory.go:184] no items to output this cycle I1122 04:21:46.413524 3798194 cpu.go:275] no items to output this cycle E1122 04:21:56.412835 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:21:56.412866 3798194 memory.go:191] Add success. W1122 04:21:56.412894 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:21:56.412905 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:21:56.412908 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:21:56.417316 3798194 cpu.go:282] Add success. I1122 04:21:56.441673 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:21:56.442180 3798194 net.go:642] Add success. I1122 04:21:56.443738 3798194 net.go:778] primary dev: ETH0 I1122 04:21:56.443752 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:21:56.443764 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:21:57.451149 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:21:57.451516 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:21:57.451531 3798194 disk_worker.go:723] disk space is not compliant W1122 04:21:57.451534 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:21:57.461057 3798194 disk_worker.go:509] system disk:vda1 I1122 04:21:57.461098 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:21:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:21:59.450120 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:21:59.450197 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:21:59.450226 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:21:59.470700 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:22:01.953482 3798194 disk_info.go:125] begin check local disk info of client I1122 04:22:01.957328 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:22:01.957337 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004db740 0xc0004db780] E1122 04:22:06.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:22:06.412252 3798194 memory.go:184] no items to output this cycle I1122 04:22:06.412399 3798194 cpu.go:275] no items to output this cycle E1122 04:22:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:22:16.412257 3798194 memory.go:184] no items to output this cycle I1122 04:22:16.412412 3798194 cpu.go:275] no items to output this cycle E1122 04:22:26.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:22:26.412274 3798194 memory.go:191] Add success. I1122 04:22:26.416639 3798194 cpu.go:282] Add success. I1122 04:22:26.442208 3798194 net.go:642] Add success. I1122 04:22:26.443793 3798194 net.go:778] primary dev: ETH0 I1122 04:22:26.443807 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:22:26.443822 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:22:29.450603 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:22:29.450684 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:22:29.450712 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:22:36.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:22:36.412274 3798194 memory.go:184] no items to output this cycle I1122 04:22:36.412427 3798194 cpu.go:275] no items to output this cycle E1122 04:22:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:22:46.413259 3798194 memory.go:184] no items to output this cycle I1122 04:22:46.413406 3798194 cpu.go:275] no items to output this cycle E1122 04:22:56.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:22:56.413264 3798194 memory.go:191] Add success. W1122 04:22:56.413300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:22:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:22:56.413315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:22:56.417417 3798194 cpu.go:282] Add success. I1122 04:22:56.441806 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:22:56.442369 3798194 net.go:642] Add success. I1122 04:22:56.443835 3798194 net.go:778] primary dev: ETH0 I1122 04:22:56.443849 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:22:56.443863 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:22:57.450668 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:22:57.450848 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:22:57.450903 3798194 disk_worker.go:723] disk space is not compliant W1122 04:22:57.450906 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:22:57.459321 3798194 disk_worker.go:509] system disk:vda1 I1122 04:22:57.459366 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:22:58.446384 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:22:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:22:59.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:22:59.450501 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:22:59.471006 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:23:01.959142 3798194 disk_info.go:125] begin check local disk info of client I1122 04:23:01.963337 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:23:01.963348 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b2c00 0xc0003b2c40] E1122 04:23:06.412475 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:23:06.412495 3798194 memory.go:184] no items to output this cycle I1122 04:23:06.412636 3798194 cpu.go:275] no items to output this cycle E1122 04:23:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:23:16.413364 3798194 memory.go:184] no items to output this cycle I1122 04:23:16.413700 3798194 cpu.go:275] no items to output this cycle E1122 04:23:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:23:26.413267 3798194 memory.go:191] Add success. I1122 04:23:26.417540 3798194 cpu.go:282] Add success. I1122 04:23:26.442237 3798194 net.go:642] Add success. I1122 04:23:26.443732 3798194 net.go:778] primary dev: ETH0 I1122 04:23:26.443747 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:23:26.443760 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:23:29.449678 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:23:29.449751 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:23:29.449780 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:23:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:23:36.412256 3798194 memory.go:184] no items to output this cycle I1122 04:23:36.412396 3798194 cpu.go:275] no items to output this cycle E1122 04:23:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:23:46.411983 3798194 memory.go:184] no items to output this cycle I1122 04:23:46.412143 3798194 cpu.go:275] no items to output this cycle E1122 04:23:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:23:56.411959 3798194 memory.go:191] Add success. W1122 04:23:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:23:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:23:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:23:56.412355 3798194 cpu.go:282] Add success. I1122 04:23:56.441918 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:23:56.442577 3798194 net.go:642] Add success. I1122 04:23:56.446753 3798194 net.go:778] primary dev: ETH0 I1122 04:23:56.446768 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:23:56.446781 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:23:57.450633 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:23:57.450846 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:23:57.450859 3798194 disk_worker.go:723] disk space is not compliant W1122 04:23:57.450862 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:23:57.459314 3798194 disk_worker.go:509] system disk:vda1 I1122 04:23:57.459353 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:23:58.445994 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:23:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:23:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:23:59.450133 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:23:59.471647 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:24:01.965146 3798194 disk_info.go:125] begin check local disk info of client I1122 04:24:01.968991 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:24:01.969000 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037c080 0xc00037c0c0] E1122 04:24:06.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:24:06.413262 3798194 memory.go:184] no items to output this cycle I1122 04:24:06.413412 3798194 cpu.go:275] no items to output this cycle E1122 04:24:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:24:16.411975 3798194 memory.go:184] no items to output this cycle I1122 04:24:16.412135 3798194 cpu.go:275] no items to output this cycle I1122 04:24:17.630560 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:24:17.630570 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:24:26.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:24:26.412261 3798194 memory.go:191] Add success. I1122 04:24:26.416518 3798194 cpu.go:282] Add success. I1122 04:24:26.442293 3798194 net.go:642] Add success. I1122 04:24:26.444152 3798194 net.go:778] primary dev: ETH0 I1122 04:24:26.444168 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:24:26.444182 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:24:29.450410 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:24:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:24:29.450511 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:24:36.413253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:24:36.413275 3798194 memory.go:184] no items to output this cycle I1122 04:24:36.413417 3798194 cpu.go:275] no items to output this cycle E1122 04:24:46.412516 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:24:46.412539 3798194 memory.go:184] no items to output this cycle I1122 04:24:46.412681 3798194 cpu.go:275] no items to output this cycle E1122 04:24:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:24:56.411961 3798194 memory.go:191] Add success. W1122 04:24:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:24:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:24:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:24:56.412400 3798194 cpu.go:282] Add success. I1122 04:24:56.440949 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:24:56.441580 3798194 net.go:642] Add success. I1122 04:24:56.445827 3798194 net.go:778] primary dev: ETH0 I1122 04:24:56.445842 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:24:56.445855 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:24:57.445557 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:24:57.445819 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:24:57.445831 3798194 disk_worker.go:723] disk space is not compliant W1122 04:24:57.445834 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:24:57.447965 3798194 disk_worker.go:509] system disk:vda1 I1122 04:24:57.448006 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:24:58.445983 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:24:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:24:59.450125 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:24:59.450161 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:24:59.471647 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:25:01.971146 3798194 disk_info.go:125] begin check local disk info of client I1122 04:25:01.974958 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:25:01.974971 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e1940 0xc0002e1980] E1122 04:25:06.413250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:25:06.413272 3798194 memory.go:184] no items to output this cycle I1122 04:25:06.413426 3798194 cpu.go:275] no items to output this cycle E1122 04:25:16.412847 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:25:16.412869 3798194 memory.go:184] no items to output this cycle I1122 04:25:16.413020 3798194 cpu.go:275] no items to output this cycle E1122 04:25:26.412177 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:25:26.412209 3798194 memory.go:191] Add success. I1122 04:25:26.416769 3798194 cpu.go:282] Add success. I1122 04:25:26.441288 3798194 net.go:642] Add success. I1122 04:25:26.442762 3798194 net.go:778] primary dev: ETH0 I1122 04:25:26.442777 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:25:26.442789 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:25:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:25:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:25:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:25:36.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:25:36.412880 3798194 memory.go:184] no items to output this cycle I1122 04:25:36.413034 3798194 cpu.go:275] no items to output this cycle E1122 04:25:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:25:46.411986 3798194 memory.go:184] no items to output this cycle I1122 04:25:46.412147 3798194 cpu.go:275] no items to output this cycle E1122 04:25:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:25:56.411949 3798194 memory.go:191] Add success. W1122 04:25:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:25:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:25:56.411991 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:25:56.412347 3798194 cpu.go:282] Add success. I1122 04:25:56.441579 3798194 event_worker.go:152] Polling the log file for events... I1122 04:25:56.444771 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:25:56.445381 3798194 net.go:642] Add success. I1122 04:25:56.449828 3798194 net.go:778] primary dev: ETH0 I1122 04:25:56.449844 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:25:56.449859 3798194 net.go:659] Add eni metrics success, count: 7 E1122 04:25:57.450605 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 04:25:57.450619 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:25:57.450838 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:25:57.450847 3798194 disk_worker.go:723] disk space is not compliant W1122 04:25:57.450850 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:25:57.459382 3798194 disk_worker.go:509] system disk:vda1 I1122 04:25:57.459426 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 04:25:58.448277 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 04:25:58.448293 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 04:25:59.448646 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 04:25:59.449720 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:25:59.449773 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:25:59.449792 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:25:59.471336 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:26:01.979476 3798194 disk_info.go:125] begin check local disk info of client I1122 04:26:01.983364 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:26:01.983372 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d42c0 0xc0003d4300] E1122 04:26:06.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:26:06.413266 3798194 memory.go:184] no items to output this cycle I1122 04:26:06.413409 3798194 cpu.go:275] no items to output this cycle E1122 04:26:16.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:26:16.411972 3798194 memory.go:184] no items to output this cycle I1122 04:26:16.412123 3798194 cpu.go:275] no items to output this cycle E1122 04:26:26.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:26:26.412264 3798194 memory.go:191] Add success. I1122 04:26:26.416710 3798194 cpu.go:282] Add success. I1122 04:26:26.441263 3798194 net.go:642] Add success. I1122 04:26:26.442719 3798194 net.go:778] primary dev: ETH0 I1122 04:26:26.442735 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:26:26.442749 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:26:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:26:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:26:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:26:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:26:36.412258 3798194 memory.go:184] no items to output this cycle I1122 04:26:36.412410 3798194 cpu.go:275] no items to output this cycle E1122 04:26:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:26:46.411977 3798194 memory.go:184] no items to output this cycle I1122 04:26:46.412138 3798194 cpu.go:275] no items to output this cycle E1122 04:26:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:26:56.411955 3798194 memory.go:191] Add success. W1122 04:26:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:26:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:26:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:26:56.412355 3798194 cpu.go:282] Add success. I1122 04:26:56.441837 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:26:56.442421 3798194 net.go:642] Add success. I1122 04:26:56.446963 3798194 net.go:778] primary dev: ETH0 I1122 04:26:56.446978 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:26:56.446991 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:26:57.450629 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:26:57.450824 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:26:57.450836 3798194 disk_worker.go:723] disk space is not compliant W1122 04:26:57.450839 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:26:57.459352 3798194 disk_worker.go:509] system disk:vda1 I1122 04:26:57.459395 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:26:58.445993 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:26:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:26:59.450125 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:26:59.450153 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:26:59.471703 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:27:01.983468 3798194 disk_info.go:125] begin check local disk info of client I1122 04:27:01.987281 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:27:01.987289 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b18c0 0xc0003b1900] E1122 04:27:06.412762 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:27:06.412783 3798194 memory.go:184] no items to output this cycle I1122 04:27:06.412932 3798194 cpu.go:275] no items to output this cycle E1122 04:27:16.413090 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:27:16.413127 3798194 memory.go:184] no items to output this cycle I1122 04:27:16.413287 3798194 cpu.go:275] no items to output this cycle I1122 04:27:17.630735 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:27:17.630743 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:27:26.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:27:26.413262 3798194 memory.go:191] Add success. I1122 04:27:26.417275 3798194 cpu.go:282] Add success. I1122 04:27:26.442151 3798194 net.go:642] Add success. I1122 04:27:26.443809 3798194 net.go:778] primary dev: ETH0 I1122 04:27:26.443823 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:27:26.443840 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:27:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:27:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:27:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:27:36.413165 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:27:36.413187 3798194 memory.go:184] no items to output this cycle I1122 04:27:36.413516 3798194 cpu.go:275] no items to output this cycle E1122 04:27:46.412836 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:27:46.412860 3798194 memory.go:184] no items to output this cycle I1122 04:27:46.413007 3798194 cpu.go:275] no items to output this cycle E1122 04:27:56.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:27:56.411983 3798194 memory.go:191] Add success. W1122 04:27:56.412013 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:27:56.412024 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:27:56.412028 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:27:56.416288 3798194 cpu.go:282] Add success. I1122 04:27:56.441657 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:27:56.442161 3798194 net.go:642] Add success. I1122 04:27:56.443707 3798194 net.go:778] primary dev: ETH0 I1122 04:27:56.443721 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:27:56.443734 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:27:57.451087 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:27:57.451282 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:27:57.451332 3798194 disk_worker.go:723] disk space is not compliant W1122 04:27:57.451335 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:27:57.459730 3798194 disk_worker.go:509] system disk:vda1 I1122 04:27:57.459771 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:27:58.446394 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:27:59.450500 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:27:59.450591 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:27:59.450618 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:27:59.471194 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:28:01.987390 3798194 disk_info.go:125] begin check local disk info of client I1122 04:28:01.991182 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:28:01.991191 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000480e00 0xc000480e40] E1122 04:28:06.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:28:06.413257 3798194 memory.go:184] no items to output this cycle I1122 04:28:06.413400 3798194 cpu.go:275] no items to output this cycle E1122 04:28:16.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:28:16.413253 3798194 memory.go:184] no items to output this cycle I1122 04:28:16.413420 3798194 cpu.go:275] no items to output this cycle E1122 04:28:26.411798 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:28:26.411830 3798194 memory.go:191] Add success. I1122 04:28:26.416307 3798194 cpu.go:282] Add success. I1122 04:28:26.441749 3798194 net.go:642] Add success. I1122 04:28:26.443185 3798194 net.go:778] primary dev: ETH0 I1122 04:28:26.443199 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:28:26.443213 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:28:29.450333 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:28:29.450412 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:28:29.450440 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:28:36.412246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:28:36.412270 3798194 memory.go:184] no items to output this cycle I1122 04:28:36.412412 3798194 cpu.go:275] no items to output this cycle E1122 04:28:46.412301 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:28:46.412323 3798194 memory.go:184] no items to output this cycle I1122 04:28:46.412473 3798194 cpu.go:275] no items to output this cycle E1122 04:28:56.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:28:56.413265 3798194 memory.go:191] Add success. W1122 04:28:56.413294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:28:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:28:56.413308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:28:56.417618 3798194 cpu.go:282] Add success. I1122 04:28:56.440957 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:28:56.441555 3798194 net.go:642] Add success. I1122 04:28:56.443080 3798194 net.go:778] primary dev: ETH0 I1122 04:28:56.443095 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:28:56.443107 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:28:57.451086 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:28:57.451269 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:28:57.451320 3798194 disk_worker.go:723] disk space is not compliant W1122 04:28:57.451323 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:28:57.460857 3798194 disk_worker.go:509] system disk:vda1 I1122 04:28:57.460897 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:28:58.446366 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:28:59.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:28:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:28:59.450502 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:28:59.471066 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:29:01.991292 3798194 disk_info.go:125] begin check local disk info of client I1122 04:29:01.995185 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:29:01.995195 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000475900 0xc000475940] E1122 04:29:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:29:06.412258 3798194 memory.go:184] no items to output this cycle I1122 04:29:06.412403 3798194 cpu.go:275] no items to output this cycle E1122 04:29:16.413095 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:29:16.413143 3798194 memory.go:184] no items to output this cycle I1122 04:29:16.413289 3798194 cpu.go:275] no items to output this cycle E1122 04:29:26.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:29:26.411947 3798194 memory.go:191] Add success. I1122 04:29:26.412314 3798194 cpu.go:282] Add success. I1122 04:29:26.441902 3798194 net.go:642] Add success. I1122 04:29:26.445986 3798194 net.go:778] primary dev: ETH0 I1122 04:29:26.446002 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:29:26.446017 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:29:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:29:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:29:29.450502 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:29:36.412096 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:29:36.412136 3798194 memory.go:184] no items to output this cycle I1122 04:29:36.412281 3798194 cpu.go:275] no items to output this cycle E1122 04:29:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:29:46.411973 3798194 memory.go:184] no items to output this cycle I1122 04:29:46.412145 3798194 cpu.go:275] no items to output this cycle E1122 04:29:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:29:56.412264 3798194 memory.go:191] Add success. W1122 04:29:56.412290 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:29:56.412301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:29:56.412305 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:29:56.416426 3798194 cpu.go:282] Add success. I1122 04:29:56.441843 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:29:56.442334 3798194 net.go:642] Add success. I1122 04:29:56.443783 3798194 net.go:778] primary dev: ETH0 I1122 04:29:56.443797 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:29:56.443808 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:29:57.450930 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:29:57.451156 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:29:57.451167 3798194 disk_worker.go:723] disk space is not compliant W1122 04:29:57.451170 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:29:57.460240 3798194 disk_worker.go:509] system disk:vda1 I1122 04:29:57.460283 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:29:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:29:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:29:59.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:29:59.450506 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:29:59.471037 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:30:01.997138 3798194 disk_info.go:125] begin check local disk info of client I1122 04:30:02.000979 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:30:02.000989 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003dab40 0xc0003dab80] E1122 04:30:06.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:30:06.411990 3798194 memory.go:184] no items to output this cycle I1122 04:30:06.412140 3798194 cpu.go:275] no items to output this cycle E1122 04:30:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:30:16.412256 3798194 memory.go:184] no items to output this cycle I1122 04:30:16.412413 3798194 cpu.go:275] no items to output this cycle I1122 04:30:17.631207 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:30:17.631215 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:30:26.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:30:26.412896 3798194 memory.go:191] Add success. I1122 04:30:26.417240 3798194 cpu.go:282] Add success. I1122 04:30:26.441749 3798194 net.go:642] Add success. I1122 04:30:26.443434 3798194 net.go:778] primary dev: ETH0 I1122 04:30:26.443450 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:30:26.443465 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:30:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:30:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:30:29.450502 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:30:36.413062 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:30:36.413083 3798194 memory.go:184] no items to output this cycle I1122 04:30:36.413251 3798194 cpu.go:275] no items to output this cycle E1122 04:30:46.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:30:46.412871 3798194 memory.go:184] no items to output this cycle I1122 04:30:46.413028 3798194 cpu.go:275] no items to output this cycle E1122 04:30:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:30:56.412266 3798194 memory.go:191] Add success. W1122 04:30:56.412294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:30:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:30:56.412308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:30:56.416286 3798194 cpu.go:282] Add success. I1122 04:30:56.444538 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:30:56.445037 3798194 net.go:642] Add success. I1122 04:30:56.446488 3798194 net.go:778] primary dev: ETH0 I1122 04:30:56.446503 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:30:56.446516 3798194 net.go:659] Add eni metrics success, count: 7 E1122 04:30:57.450964 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 04:30:57.450978 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:30:57.451158 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:30:57.451212 3798194 disk_worker.go:723] disk space is not compliant W1122 04:30:57.451215 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:30:57.460079 3798194 disk_worker.go:509] system disk:vda1 I1122 04:30:57.460142 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 04:30:58.447833 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 04:30:58.447848 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 04:30:59.448934 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 04:30:59.450008 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:30:59.450057 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:30:59.450076 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:30:59.471632 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:31:02.001083 3798194 disk_info.go:125] begin check local disk info of client I1122 04:31:02.004991 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:31:02.005004 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000326000 0xc000326040] E1122 04:31:06.411940 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:31:06.411964 3798194 memory.go:184] no items to output this cycle I1122 04:31:06.412106 3798194 cpu.go:275] no items to output this cycle E1122 04:31:16.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:31:16.412253 3798194 memory.go:184] no items to output this cycle I1122 04:31:16.412406 3798194 cpu.go:275] no items to output this cycle E1122 04:31:26.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:31:26.412276 3798194 memory.go:191] Add success. I1122 04:31:26.416411 3798194 cpu.go:282] Add success. I1122 04:31:26.441957 3798194 net.go:642] Add success. I1122 04:31:26.443407 3798194 net.go:778] primary dev: ETH0 I1122 04:31:26.443422 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:31:26.443436 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:31:29.450118 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:31:29.450196 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:31:29.450220 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:31:36.412419 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:31:36.412445 3798194 memory.go:184] no items to output this cycle I1122 04:31:36.412608 3798194 cpu.go:275] no items to output this cycle E1122 04:31:46.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:31:46.412874 3798194 memory.go:184] no items to output this cycle I1122 04:31:46.413028 3798194 cpu.go:275] no items to output this cycle E1122 04:31:56.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:31:56.413262 3798194 memory.go:191] Add success. W1122 04:31:56.413295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:31:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:31:56.413310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:31:56.417510 3798194 cpu.go:282] Add success. I1122 04:31:56.441819 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:31:56.442443 3798194 net.go:642] Add success. I1122 04:31:56.443945 3798194 net.go:778] primary dev: ETH0 I1122 04:31:56.443960 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:31:56.443972 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:31:57.445583 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:31:57.445856 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:31:57.445869 3798194 disk_worker.go:723] disk space is not compliant W1122 04:31:57.445872 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:31:57.450882 3798194 disk_worker.go:509] system disk:vda1 I1122 04:31:57.450926 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:31:58.445987 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:31:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:31:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:31:59.450149 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:31:59.471705 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:32:02.007148 3798194 disk_info.go:125] begin check local disk info of client I1122 04:32:02.010979 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:32:02.010989 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030f400 0xc00030f440] E1122 04:32:06.411944 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:32:06.411967 3798194 memory.go:184] no items to output this cycle I1122 04:32:06.412118 3798194 cpu.go:275] no items to output this cycle E1122 04:32:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:32:16.413262 3798194 memory.go:184] no items to output this cycle I1122 04:32:16.413411 3798194 cpu.go:275] no items to output this cycle E1122 04:32:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:32:26.411952 3798194 memory.go:191] Add success. I1122 04:32:26.412285 3798194 cpu.go:282] Add success. I1122 04:32:26.441938 3798194 net.go:642] Add success. I1122 04:32:26.446322 3798194 net.go:778] primary dev: ETH0 I1122 04:32:26.446337 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:32:26.446350 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:32:29.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:32:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:32:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:32:36.412170 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:32:36.412192 3798194 memory.go:184] no items to output this cycle I1122 04:32:36.412340 3798194 cpu.go:275] no items to output this cycle E1122 04:32:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:32:46.413255 3798194 memory.go:184] no items to output this cycle I1122 04:32:46.413410 3798194 cpu.go:275] no items to output this cycle E1122 04:32:56.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:32:56.412276 3798194 memory.go:191] Add success. W1122 04:32:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:32:56.412321 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:32:56.412324 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:32:56.416415 3798194 cpu.go:282] Add success. I1122 04:32:56.441804 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:32:56.442396 3798194 net.go:642] Add success. I1122 04:32:56.443794 3798194 net.go:778] primary dev: ETH0 I1122 04:32:56.443809 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:32:56.443820 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:32:57.450824 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:32:57.450983 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:32:57.451040 3798194 disk_worker.go:723] disk space is not compliant W1122 04:32:57.451044 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:32:57.459377 3798194 disk_worker.go:509] system disk:vda1 I1122 04:32:57.459419 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:32:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:32:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:32:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:32:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:32:59.471667 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:33:02.013137 3798194 disk_info.go:125] begin check local disk info of client I1122 04:33:02.016949 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:33:02.016958 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000466e40 0xc000466e80] E1122 04:33:06.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:33:06.412872 3798194 memory.go:184] no items to output this cycle I1122 04:33:06.413013 3798194 cpu.go:275] no items to output this cycle E1122 04:33:16.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:33:16.412269 3798194 memory.go:184] no items to output this cycle I1122 04:33:16.412416 3798194 cpu.go:275] no items to output this cycle I1122 04:33:17.631375 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:33:17.631383 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:33:26.411352 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:33:26.411386 3798194 memory.go:191] Add success. I1122 04:33:26.411736 3798194 cpu.go:282] Add success. I1122 04:33:26.441288 3798194 net.go:642] Add success. I1122 04:33:26.445238 3798194 net.go:778] primary dev: ETH0 I1122 04:33:26.445253 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:33:26.445266 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:33:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:33:29.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:33:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:33:36.412407 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:33:36.412433 3798194 memory.go:184] no items to output this cycle I1122 04:33:36.412578 3798194 cpu.go:275] no items to output this cycle E1122 04:33:46.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:33:46.412882 3798194 memory.go:184] no items to output this cycle I1122 04:33:46.413029 3798194 cpu.go:275] no items to output this cycle E1122 04:33:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:33:56.411956 3798194 memory.go:191] Add success. W1122 04:33:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:33:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:33:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:33:56.412360 3798194 cpu.go:282] Add success. I1122 04:33:56.441815 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:33:56.442339 3798194 net.go:642] Add success. I1122 04:33:56.452759 3798194 net.go:778] primary dev: ETH0 I1122 04:33:56.452775 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:33:56.452790 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:33:57.451043 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:33:57.451290 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:33:57.451301 3798194 disk_worker.go:723] disk space is not compliant W1122 04:33:57.451304 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:33:57.459755 3798194 disk_worker.go:509] system disk:vda1 I1122 04:33:57.459795 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:33:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:33:59.450326 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:33:59.450414 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:33:59.450441 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:33:59.470995 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:34:02.019149 3798194 disk_info.go:125] begin check local disk info of client I1122 04:34:02.022911 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:34:02.022920 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000398380 0xc0003983c0] E1122 04:34:06.412078 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:34:06.412101 3798194 memory.go:184] no items to output this cycle I1122 04:34:06.412266 3798194 cpu.go:275] no items to output this cycle E1122 04:34:16.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:34:16.413259 3798194 memory.go:184] no items to output this cycle I1122 04:34:16.413406 3798194 cpu.go:275] no items to output this cycle E1122 04:34:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:34:26.413277 3798194 memory.go:191] Add success. I1122 04:34:26.417169 3798194 cpu.go:282] Add success. I1122 04:34:26.441679 3798194 net.go:642] Add success. I1122 04:34:26.443417 3798194 net.go:778] primary dev: ETH0 I1122 04:34:26.443440 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:34:26.443452 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:34:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:34:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:34:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:34:36.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:34:36.412882 3798194 memory.go:184] no items to output this cycle I1122 04:34:36.413026 3798194 cpu.go:275] no items to output this cycle E1122 04:34:46.411777 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:34:46.411799 3798194 memory.go:184] no items to output this cycle I1122 04:34:46.412163 3798194 cpu.go:275] no items to output this cycle E1122 04:34:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:34:56.411945 3798194 memory.go:191] Add success. W1122 04:34:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:34:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:34:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:34:56.412358 3798194 cpu.go:282] Add success. I1122 04:34:56.441812 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:34:56.442318 3798194 net.go:642] Add success. I1122 04:34:56.446574 3798194 net.go:778] primary dev: ETH0 I1122 04:34:56.446590 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:34:56.446603 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:34:57.450930 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:34:57.451165 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:34:57.451177 3798194 disk_worker.go:723] disk space is not compliant W1122 04:34:57.451180 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:34:57.460102 3798194 disk_worker.go:509] system disk:vda1 I1122 04:34:57.460161 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:34:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:34:59.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:34:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:34:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:34:59.471655 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:35:02.025143 3798194 disk_info.go:125] begin check local disk info of client I1122 04:35:02.028968 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:35:02.028976 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00043c500 0xc00043c540] E1122 04:35:06.413165 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:35:06.413189 3798194 memory.go:184] no items to output this cycle I1122 04:35:06.413332 3798194 cpu.go:275] no items to output this cycle E1122 04:35:16.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:35:16.413262 3798194 memory.go:184] no items to output this cycle I1122 04:35:16.413406 3798194 cpu.go:275] no items to output this cycle E1122 04:35:26.411927 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:35:26.411957 3798194 memory.go:191] Add success. I1122 04:35:26.412308 3798194 cpu.go:282] Add success. I1122 04:35:26.441935 3798194 net.go:642] Add success. I1122 04:35:26.445843 3798194 net.go:778] primary dev: ETH0 I1122 04:35:26.445867 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:35:26.445880 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:35:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:35:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:35:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:35:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:35:36.412253 3798194 memory.go:184] no items to output this cycle I1122 04:35:36.412400 3798194 cpu.go:275] no items to output this cycle E1122 04:35:46.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:35:46.412259 3798194 memory.go:184] no items to output this cycle I1122 04:35:46.412407 3798194 cpu.go:275] no items to output this cycle E1122 04:35:56.412317 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:35:56.412351 3798194 memory.go:191] Add success. W1122 04:35:56.412379 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:35:56.412391 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:35:56.412394 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:35:56.416482 3798194 cpu.go:282] Add success. I1122 04:35:56.441632 3798194 event_worker.go:152] Polling the log file for events... I1122 04:35:56.444867 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:35:56.445395 3798194 net.go:642] Add success. I1122 04:35:56.446808 3798194 net.go:778] primary dev: ETH0 I1122 04:35:56.446822 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:35:56.446835 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:35:56.446852 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795291556, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 04:35:56.454506 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 04:35:57.449163 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 04:35:57.449178 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:35:57.449399 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:35:57.449409 3798194 disk_worker.go:723] disk space is not compliant W1122 04:35:57.449412 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:35:57.451952 3798194 disk_worker.go:509] system disk:vda1 I1122 04:35:57.451990 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 04:35:58.448138 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 04:35:58.448153 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 04:35:59.448627 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 04:35:59.449660 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:35:59.449710 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:35:59.449735 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:35:59.471186 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:36:02.031483 3798194 disk_info.go:125] begin check local disk info of client I1122 04:36:02.035361 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:36:02.035369 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000489580 0xc0004895c0] E1122 04:36:06.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:36:06.413252 3798194 memory.go:184] no items to output this cycle I1122 04:36:06.413393 3798194 cpu.go:275] no items to output this cycle E1122 04:36:16.411973 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:36:16.411996 3798194 memory.go:184] no items to output this cycle I1122 04:36:16.412157 3798194 cpu.go:275] no items to output this cycle I1122 04:36:17.631537 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:36:17.631545 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:36:26.412011 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:36:26.412048 3798194 memory.go:191] Add success. I1122 04:36:26.416250 3798194 cpu.go:282] Add success. I1122 04:36:26.441757 3798194 net.go:642] Add success. I1122 04:36:26.443159 3798194 net.go:778] primary dev: ETH0 I1122 04:36:26.443174 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:36:26.443189 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:36:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:36:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:36:29.450147 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:36:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:36:36.411986 3798194 memory.go:184] no items to output this cycle I1122 04:36:36.412139 3798194 cpu.go:275] no items to output this cycle E1122 04:36:46.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:36:46.411967 3798194 memory.go:184] no items to output this cycle I1122 04:36:46.412132 3798194 cpu.go:275] no items to output this cycle E1122 04:36:56.413223 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:36:56.413257 3798194 memory.go:191] Add success. W1122 04:36:56.413282 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:36:56.413296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:36:56.413299 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:36:56.417610 3798194 cpu.go:282] Add success. I1122 04:36:56.440940 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:36:56.441494 3798194 net.go:642] Add success. I1122 04:36:56.443050 3798194 net.go:778] primary dev: ETH0 I1122 04:36:56.443065 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:36:56.443077 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:36:57.450932 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:36:57.451155 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:36:57.451167 3798194 disk_worker.go:723] disk space is not compliant W1122 04:36:57.451170 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:36:57.460020 3798194 disk_worker.go:509] system disk:vda1 I1122 04:36:57.460064 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:36:58.446364 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:36:59.450386 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:36:59.450465 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:36:59.450492 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:36:59.471024 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:37:02.038138 3798194 disk_info.go:125] begin check local disk info of client I1122 04:37:02.041959 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:37:02.041968 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d8dc0 0xc0003d8e00] E1122 04:37:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:37:06.413256 3798194 memory.go:184] no items to output this cycle I1122 04:37:06.413397 3798194 cpu.go:275] no items to output this cycle E1122 04:37:16.412869 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:37:16.412891 3798194 memory.go:184] no items to output this cycle I1122 04:37:16.413041 3798194 cpu.go:275] no items to output this cycle E1122 04:37:26.413038 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:37:26.413077 3798194 memory.go:191] Add success. I1122 04:37:26.417081 3798194 cpu.go:282] Add success. I1122 04:37:26.441587 3798194 net.go:642] Add success. I1122 04:37:26.442903 3798194 net.go:778] primary dev: ETH0 I1122 04:37:26.442917 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:37:26.442930 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:37:29.449681 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:37:29.449765 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:37:29.449794 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:37:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:37:36.411979 3798194 memory.go:184] no items to output this cycle I1122 04:37:36.412134 3798194 cpu.go:275] no items to output this cycle E1122 04:37:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:37:46.411979 3798194 memory.go:184] no items to output this cycle I1122 04:37:46.412141 3798194 cpu.go:275] no items to output this cycle E1122 04:37:56.412273 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:37:56.412312 3798194 memory.go:191] Add success. W1122 04:37:56.412346 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:37:56.412358 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:37:56.412361 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:37:56.416440 3798194 cpu.go:282] Add success. I1122 04:37:56.441796 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:37:56.442314 3798194 net.go:642] Add success. I1122 04:37:56.443740 3798194 net.go:778] primary dev: ETH0 I1122 04:37:56.443754 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:37:56.443767 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:37:57.451533 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:37:57.451764 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:37:57.451776 3798194 disk_worker.go:723] disk space is not compliant W1122 04:37:57.451779 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:37:57.460191 3798194 disk_worker.go:509] system disk:vda1 I1122 04:37:57.460233 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:37:58.446012 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:37:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:37:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:37:59.450146 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:37:59.471695 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:38:02.044141 3798194 disk_info.go:125] begin check local disk info of client I1122 04:38:02.047974 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:38:02.047983 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d9000 0xc0003d9040] E1122 04:38:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:38:06.411976 3798194 memory.go:184] no items to output this cycle I1122 04:38:06.412135 3798194 cpu.go:275] no items to output this cycle E1122 04:38:16.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:38:16.413253 3798194 memory.go:184] no items to output this cycle I1122 04:38:16.413399 3798194 cpu.go:275] no items to output this cycle E1122 04:38:26.412874 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:38:26.412908 3798194 memory.go:191] Add success. I1122 04:38:26.417076 3798194 cpu.go:282] Add success. I1122 04:38:26.441618 3798194 net.go:642] Add success. I1122 04:38:26.443073 3798194 net.go:778] primary dev: ETH0 I1122 04:38:26.443088 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:38:26.443102 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:38:29.450039 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:38:29.450125 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:38:29.450151 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:38:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:38:36.411979 3798194 memory.go:184] no items to output this cycle I1122 04:38:36.412139 3798194 cpu.go:275] no items to output this cycle E1122 04:38:46.412249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:38:46.412272 3798194 memory.go:184] no items to output this cycle I1122 04:38:46.412540 3798194 cpu.go:275] no items to output this cycle E1122 04:38:56.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:38:56.411964 3798194 memory.go:191] Add success. W1122 04:38:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:38:56.412011 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:38:56.412015 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:38:56.412363 3798194 cpu.go:282] Add success. I1122 04:38:56.441822 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:38:56.442407 3798194 net.go:642] Add success. I1122 04:38:56.446563 3798194 net.go:778] primary dev: ETH0 I1122 04:38:56.446579 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:38:56.446592 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:38:57.451311 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:38:57.451477 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:38:57.451531 3798194 disk_worker.go:723] disk space is not compliant W1122 04:38:57.451535 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:38:57.464690 3798194 disk_worker.go:509] system disk:vda1 I1122 04:38:57.464742 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:38:58.446377 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:38:59.450621 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:38:59.450695 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:38:59.450723 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:38:59.471184 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:39:02.050144 3798194 disk_info.go:125] begin check local disk info of client I1122 04:39:02.053940 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:39:02.053949 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00045ee40 0xc00045ee80] E1122 04:39:06.412975 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:39:06.412996 3798194 memory.go:184] no items to output this cycle I1122 04:39:06.413168 3798194 cpu.go:275] no items to output this cycle E1122 04:39:16.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:39:16.412251 3798194 memory.go:184] no items to output this cycle I1122 04:39:16.412400 3798194 cpu.go:275] no items to output this cycle I1122 04:39:17.632208 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:39:17.632217 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:39:26.411642 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:39:26.411675 3798194 memory.go:191] Add success. I1122 04:39:26.412009 3798194 cpu.go:282] Add success. I1122 04:39:26.441642 3798194 net.go:642] Add success. I1122 04:39:26.445873 3798194 net.go:778] primary dev: ETH0 I1122 04:39:26.445888 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:39:26.445900 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:39:29.450462 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:39:29.450545 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:39:29.450570 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:39:36.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:39:36.411973 3798194 memory.go:184] no items to output this cycle I1122 04:39:36.412126 3798194 cpu.go:275] no items to output this cycle E1122 04:39:46.413611 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:39:46.413646 3798194 memory.go:184] no items to output this cycle I1122 04:39:46.413797 3798194 cpu.go:275] no items to output this cycle E1122 04:39:56.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:39:56.413271 3798194 memory.go:191] Add success. W1122 04:39:56.413299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:39:56.413310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:39:56.413314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:39:56.417464 3798194 cpu.go:282] Add success. I1122 04:39:56.440941 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:39:56.441536 3798194 net.go:642] Add success. I1122 04:39:56.443103 3798194 net.go:778] primary dev: ETH0 I1122 04:39:56.443127 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:39:56.443140 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:39:57.450995 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:39:57.451179 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:39:57.451233 3798194 disk_worker.go:723] disk space is not compliant W1122 04:39:57.451236 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:39:57.460418 3798194 disk_worker.go:509] system disk:vda1 I1122 04:39:57.460455 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:39:58.446368 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:39:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:39:59.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:39:59.450507 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:39:59.471070 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:40:02.054056 3798194 disk_info.go:125] begin check local disk info of client I1122 04:40:02.057898 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:40:02.057908 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000447440 0xc000447480] E1122 04:40:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:40:06.412260 3798194 memory.go:184] no items to output this cycle I1122 04:40:06.412409 3798194 cpu.go:275] no items to output this cycle E1122 04:40:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:40:16.412262 3798194 memory.go:184] no items to output this cycle I1122 04:40:16.412406 3798194 cpu.go:275] no items to output this cycle E1122 04:40:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:40:26.411959 3798194 memory.go:191] Add success. I1122 04:40:26.412305 3798194 cpu.go:282] Add success. I1122 04:40:26.442069 3798194 net.go:642] Add success. I1122 04:40:26.446654 3798194 net.go:778] primary dev: ETH0 I1122 04:40:26.446670 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:40:26.446683 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:40:29.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:40:29.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:40:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:40:36.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:40:36.413255 3798194 memory.go:184] no items to output this cycle I1122 04:40:36.413400 3798194 cpu.go:275] no items to output this cycle E1122 04:40:46.413450 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:40:46.413471 3798194 memory.go:184] no items to output this cycle I1122 04:40:46.413620 3798194 cpu.go:275] no items to output this cycle E1122 04:40:56.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:40:56.412263 3798194 memory.go:191] Add success. W1122 04:40:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:40:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:40:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:40:56.419306 3798194 cpu.go:282] Add success. I1122 04:40:56.444998 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:40:56.445569 3798194 net.go:642] Add success. I1122 04:40:56.447096 3798194 net.go:778] primary dev: ETH0 I1122 04:40:56.447120 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:40:56.447134 3798194 net.go:659] Add eni metrics success, count: 7 E1122 04:40:57.450698 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 04:40:57.450711 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:40:57.450876 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:40:57.450930 3798194 disk_worker.go:723] disk space is not compliant W1122 04:40:57.450933 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:40:57.459353 3798194 disk_worker.go:509] system disk:vda1 I1122 04:40:57.459393 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 04:40:58.448075 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 04:40:58.448088 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 04:40:59.448629 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 04:40:59.449704 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:40:59.449755 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:40:59.449774 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:40:59.471306 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:41:02.061010 3798194 disk_info.go:125] begin check local disk info of client I1122 04:41:02.064814 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:41:02.064823 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ea4c0 0xc0003ea500] E1122 04:41:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:41:06.411974 3798194 memory.go:184] no items to output this cycle I1122 04:41:06.412130 3798194 cpu.go:275] no items to output this cycle E1122 04:41:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:41:16.411976 3798194 memory.go:184] no items to output this cycle I1122 04:41:16.412132 3798194 cpu.go:275] no items to output this cycle E1122 04:41:26.413052 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:41:26.413085 3798194 memory.go:191] Add success. I1122 04:41:26.417778 3798194 cpu.go:282] Add success. I1122 04:41:26.442152 3798194 net.go:642] Add success. I1122 04:41:26.443605 3798194 net.go:778] primary dev: ETH0 I1122 04:41:26.443621 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:41:26.443633 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:41:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:41:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:41:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:41:36.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:41:36.413252 3798194 memory.go:184] no items to output this cycle I1122 04:41:36.413409 3798194 cpu.go:275] no items to output this cycle E1122 04:41:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:41:46.411980 3798194 memory.go:184] no items to output this cycle I1122 04:41:46.412140 3798194 cpu.go:275] no items to output this cycle E1122 04:41:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:41:56.411948 3798194 memory.go:191] Add success. W1122 04:41:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:41:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:41:56.411993 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:41:56.412353 3798194 cpu.go:282] Add success. I1122 04:41:56.441799 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:41:56.442317 3798194 net.go:642] Add success. I1122 04:41:56.446099 3798194 net.go:778] primary dev: ETH0 I1122 04:41:56.446127 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:41:56.446141 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:41:57.450398 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:41:57.450607 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:41:57.450617 3798194 disk_worker.go:723] disk space is not compliant W1122 04:41:57.450621 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:41:57.458984 3798194 disk_worker.go:509] system disk:vda1 I1122 04:41:57.459026 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:41:58.445635 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:41:59.449976 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:41:59.450063 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:41:59.450089 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:41:59.471445 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:42:02.064923 3798194 disk_info.go:125] begin check local disk info of client I1122 04:42:02.068734 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:42:02.068743 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00026c880 0xc00026c8c0] E1122 04:42:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:42:06.412253 3798194 memory.go:184] no items to output this cycle I1122 04:42:06.412400 3798194 cpu.go:275] no items to output this cycle E1122 04:42:16.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:42:16.412252 3798194 memory.go:184] no items to output this cycle I1122 04:42:16.412399 3798194 cpu.go:275] no items to output this cycle I1122 04:42:17.632379 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:42:17.632387 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:42:26.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:42:26.412260 3798194 memory.go:191] Add success. I1122 04:42:26.416718 3798194 cpu.go:282] Add success. I1122 04:42:26.441615 3798194 net.go:642] Add success. I1122 04:42:26.443079 3798194 net.go:778] primary dev: ETH0 I1122 04:42:26.443093 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:42:26.443108 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:42:29.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:42:29.450466 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:42:29.450493 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:42:36.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:42:36.413251 3798194 memory.go:184] no items to output this cycle I1122 04:42:36.413399 3798194 cpu.go:275] no items to output this cycle E1122 04:42:46.413757 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:42:46.413779 3798194 memory.go:184] no items to output this cycle I1122 04:42:46.413929 3798194 cpu.go:275] no items to output this cycle E1122 04:42:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:42:56.411949 3798194 memory.go:191] Add success. W1122 04:42:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:42:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:42:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:42:56.412353 3798194 cpu.go:282] Add success. I1122 04:42:56.440984 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:42:56.441591 3798194 net.go:642] Add success. I1122 04:42:56.445753 3798194 net.go:778] primary dev: ETH0 I1122 04:42:56.445768 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:42:56.445781 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:42:57.451008 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:42:57.451231 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:42:57.451243 3798194 disk_worker.go:723] disk space is not compliant W1122 04:42:57.451246 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:42:57.459978 3798194 disk_worker.go:509] system disk:vda1 I1122 04:42:57.460022 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:42:58.445994 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:42:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:42:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:42:59.450133 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:42:59.471663 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:43:02.071146 3798194 disk_info.go:125] begin check local disk info of client I1122 04:43:02.074937 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:43:02.074946 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071c80 0xc000071cc0] E1122 04:43:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:43:06.411978 3798194 memory.go:184] no items to output this cycle I1122 04:43:06.412137 3798194 cpu.go:275] no items to output this cycle E1122 04:43:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:43:16.413255 3798194 memory.go:184] no items to output this cycle I1122 04:43:16.413404 3798194 cpu.go:275] no items to output this cycle E1122 04:43:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:43:26.413265 3798194 memory.go:191] Add success. I1122 04:43:26.417989 3798194 cpu.go:282] Add success. I1122 04:43:26.441858 3798194 net.go:642] Add success. I1122 04:43:26.443344 3798194 net.go:778] primary dev: ETH0 I1122 04:43:26.443358 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:43:26.443370 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:43:29.449668 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:43:29.449745 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:43:29.449770 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:43:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:43:36.411977 3798194 memory.go:184] no items to output this cycle I1122 04:43:36.412156 3798194 cpu.go:275] no items to output this cycle E1122 04:43:46.413410 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:43:46.413433 3798194 memory.go:184] no items to output this cycle I1122 04:43:46.413579 3798194 cpu.go:275] no items to output this cycle E1122 04:43:56.412844 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:43:56.412879 3798194 memory.go:191] Add success. W1122 04:43:56.412907 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:43:56.412919 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:43:56.412922 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:43:56.417522 3798194 cpu.go:282] Add success. I1122 04:43:56.440949 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:43:56.441535 3798194 net.go:642] Add success. I1122 04:43:56.443140 3798194 net.go:778] primary dev: ETH0 I1122 04:43:56.443155 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:43:56.443168 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:43:57.450461 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:43:57.450658 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:43:57.450713 3798194 disk_worker.go:723] disk space is not compliant W1122 04:43:57.450717 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:43:57.459094 3798194 disk_worker.go:509] system disk:vda1 I1122 04:43:57.459148 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:43:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:43:59.450007 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:43:59.450088 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:43:59.450128 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:43:59.471689 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:44:02.077136 3798194 disk_info.go:125] begin check local disk info of client I1122 04:44:02.080934 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:44:02.080943 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00043ea00 0xc00043ea40] E1122 04:44:06.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:44:06.412263 3798194 memory.go:184] no items to output this cycle I1122 04:44:06.412407 3798194 cpu.go:275] no items to output this cycle E1122 04:44:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:44:16.411985 3798194 memory.go:184] no items to output this cycle I1122 04:44:16.412143 3798194 cpu.go:275] no items to output this cycle E1122 04:44:26.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:44:26.413286 3798194 memory.go:191] Add success. I1122 04:44:26.417845 3798194 cpu.go:282] Add success. I1122 04:44:26.441416 3798194 net.go:642] Add success. I1122 04:44:26.442822 3798194 net.go:778] primary dev: ETH0 I1122 04:44:26.442836 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:44:26.442850 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:44:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:44:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:44:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:44:36.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:44:36.412886 3798194 memory.go:184] no items to output this cycle I1122 04:44:36.413029 3798194 cpu.go:275] no items to output this cycle E1122 04:44:46.413293 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:44:46.413314 3798194 memory.go:184] no items to output this cycle I1122 04:44:46.413465 3798194 cpu.go:275] no items to output this cycle E1122 04:44:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:44:56.411949 3798194 memory.go:191] Add success. W1122 04:44:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:44:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:44:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:44:56.412345 3798194 cpu.go:282] Add success. I1122 04:44:56.441835 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:44:56.442405 3798194 net.go:642] Add success. I1122 04:44:56.446366 3798194 net.go:778] primary dev: ETH0 I1122 04:44:56.446383 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:44:56.446398 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:44:57.451062 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:44:57.451297 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:44:57.451308 3798194 disk_worker.go:723] disk space is not compliant W1122 04:44:57.451311 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:44:57.459834 3798194 disk_worker.go:509] system disk:vda1 I1122 04:44:57.459877 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:44:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:44:59.450008 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:44:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:44:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:44:59.471673 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:45:02.084148 3798194 disk_info.go:125] begin check local disk info of client I1122 04:45:02.087945 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:45:02.087955 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000396140 0xc000396180] E1122 04:45:06.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:45:06.412892 3798194 memory.go:184] no items to output this cycle I1122 04:45:06.413038 3798194 cpu.go:275] no items to output this cycle E1122 04:45:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:45:16.412262 3798194 memory.go:184] no items to output this cycle I1122 04:45:16.412421 3798194 cpu.go:275] no items to output this cycle I1122 04:45:17.632551 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:45:17.632559 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:45:26.411133 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:45:26.411165 3798194 memory.go:191] Add success. I1122 04:45:26.411499 3798194 cpu.go:282] Add success. I1122 04:45:26.442186 3798194 net.go:642] Add success. I1122 04:45:26.446145 3798194 net.go:778] primary dev: ETH0 I1122 04:45:26.446161 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:45:26.446177 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:45:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:45:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:45:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:45:36.412819 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:45:36.412847 3798194 memory.go:184] no items to output this cycle I1122 04:45:36.412992 3798194 cpu.go:275] no items to output this cycle E1122 04:45:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:45:46.411977 3798194 memory.go:184] no items to output this cycle I1122 04:45:46.412235 3798194 cpu.go:275] no items to output this cycle E1122 04:45:56.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:45:56.411959 3798194 memory.go:191] Add success. W1122 04:45:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:45:56.412003 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:45:56.412007 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:45:56.412379 3798194 cpu.go:282] Add success. I1122 04:45:56.441571 3798194 event_worker.go:152] Polling the log file for events... I1122 04:45:56.444735 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:45:56.445349 3798194 net.go:642] Add success. I1122 04:45:56.449511 3798194 net.go:778] primary dev: ETH0 I1122 04:45:56.449527 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:45:56.449539 3798194 net.go:659] Add eni metrics success, count: 7 E1122 04:45:57.450663 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 04:45:57.450675 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:45:57.450892 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:45:57.450902 3798194 disk_worker.go:723] disk space is not compliant W1122 04:45:57.450905 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:45:57.459209 3798194 disk_worker.go:509] system disk:vda1 I1122 04:45:57.459250 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 04:45:58.448129 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 04:45:58.448144 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 04:45:59.448614 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 04:45:59.449692 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:45:59.449754 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:45:59.449773 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:45:59.471271 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:46:02.091149 3798194 disk_info.go:125] begin check local disk info of client I1122 04:46:02.095155 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:46:02.095165 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000462440 0xc000462480] E1122 04:46:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:46:06.411981 3798194 memory.go:184] no items to output this cycle I1122 04:46:06.412139 3798194 cpu.go:275] no items to output this cycle E1122 04:46:16.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:46:16.411988 3798194 memory.go:184] no items to output this cycle I1122 04:46:16.412146 3798194 cpu.go:275] no items to output this cycle E1122 04:46:26.413510 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:46:26.413543 3798194 memory.go:191] Add success. I1122 04:46:26.417919 3798194 cpu.go:282] Add success. I1122 04:46:26.441532 3798194 net.go:642] Add success. I1122 04:46:26.443130 3798194 net.go:778] primary dev: ETH0 I1122 04:46:26.443147 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:46:26.443161 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:46:29.450453 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:46:29.450536 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:46:29.450562 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:46:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:46:36.411978 3798194 memory.go:184] no items to output this cycle I1122 04:46:36.412127 3798194 cpu.go:275] no items to output this cycle E1122 04:46:46.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:46:46.412259 3798194 memory.go:184] no items to output this cycle I1122 04:46:46.412407 3798194 cpu.go:275] no items to output this cycle E1122 04:46:56.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:46:56.412887 3798194 memory.go:191] Add success. W1122 04:46:56.412920 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:46:56.412932 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:46:56.412935 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:46:56.417011 3798194 cpu.go:282] Add success. I1122 04:46:56.441376 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:46:56.441914 3798194 net.go:642] Add success. I1122 04:46:56.443346 3798194 net.go:778] primary dev: ETH0 I1122 04:46:56.443363 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:46:56.443376 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:46:57.450717 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:46:57.450876 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:46:57.450932 3798194 disk_worker.go:723] disk space is not compliant W1122 04:46:57.450935 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:46:57.459259 3798194 disk_worker.go:509] system disk:vda1 I1122 04:46:57.459312 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:46:58.445683 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:46:59.449666 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:46:59.449751 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:46:59.449780 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:46:59.471318 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:47:02.095269 3798194 disk_info.go:125] begin check local disk info of client I1122 04:47:02.099090 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:47:02.099099 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000492640 0xc000492680] E1122 04:47:06.412786 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:47:06.412808 3798194 memory.go:184] no items to output this cycle I1122 04:47:06.412952 3798194 cpu.go:275] no items to output this cycle E1122 04:47:16.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:47:16.412270 3798194 memory.go:184] no items to output this cycle I1122 04:47:16.412412 3798194 cpu.go:275] no items to output this cycle E1122 04:47:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:47:26.411956 3798194 memory.go:191] Add success. I1122 04:47:26.412304 3798194 cpu.go:282] Add success. I1122 04:47:26.441912 3798194 net.go:642] Add success. I1122 04:47:26.445987 3798194 net.go:778] primary dev: ETH0 I1122 04:47:26.446002 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:47:26.446016 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:47:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:47:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:47:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:47:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:47:36.413261 3798194 memory.go:184] no items to output this cycle I1122 04:47:36.413406 3798194 cpu.go:275] no items to output this cycle E1122 04:47:46.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:47:46.412266 3798194 memory.go:184] no items to output this cycle I1122 04:47:46.412431 3798194 cpu.go:275] no items to output this cycle E1122 04:47:56.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:47:56.412260 3798194 memory.go:191] Add success. W1122 04:47:56.412287 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:47:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:47:56.412302 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:47:56.416441 3798194 cpu.go:282] Add success. I1122 04:47:56.441828 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:47:56.442350 3798194 net.go:642] Add success. I1122 04:47:56.443976 3798194 net.go:778] primary dev: ETH0 I1122 04:47:56.443990 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:47:56.444003 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:47:57.451298 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:47:57.451498 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:47:57.451509 3798194 disk_worker.go:723] disk space is not compliant W1122 04:47:57.451512 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:47:57.460023 3798194 disk_worker.go:509] system disk:vda1 I1122 04:47:57.460064 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:47:58.446377 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:47:59.450515 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:47:59.450598 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:47:59.450624 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:47:59.471129 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:48:02.102214 3798194 disk_info.go:125] begin check local disk info of client I1122 04:48:02.106050 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:48:02.106060 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003957c0 0xc000395800] E1122 04:48:06.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:48:06.412265 3798194 memory.go:184] no items to output this cycle I1122 04:48:06.412408 3798194 cpu.go:275] no items to output this cycle E1122 04:48:16.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:48:16.412254 3798194 memory.go:184] no items to output this cycle I1122 04:48:16.412401 3798194 cpu.go:275] no items to output this cycle I1122 04:48:17.632722 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:48:17.632730 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:48:26.412246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:48:26.412278 3798194 memory.go:191] Add success. I1122 04:48:26.416392 3798194 cpu.go:282] Add success. I1122 04:48:26.442067 3798194 net.go:642] Add success. I1122 04:48:26.443537 3798194 net.go:778] primary dev: ETH0 I1122 04:48:26.443552 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:48:26.443566 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:48:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:48:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:48:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:48:36.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:48:36.413259 3798194 memory.go:184] no items to output this cycle I1122 04:48:36.413402 3798194 cpu.go:275] no items to output this cycle E1122 04:48:46.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:48:46.411991 3798194 memory.go:184] no items to output this cycle I1122 04:48:46.412145 3798194 cpu.go:275] no items to output this cycle E1122 04:48:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:48:56.411948 3798194 memory.go:191] Add success. W1122 04:48:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:48:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:48:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:48:56.412339 3798194 cpu.go:282] Add success. I1122 04:48:56.441934 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:48:56.442497 3798194 net.go:642] Add success. I1122 04:48:56.446580 3798194 net.go:778] primary dev: ETH0 I1122 04:48:56.446594 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:48:56.446607 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:48:57.450704 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:48:57.450857 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:48:57.450911 3798194 disk_worker.go:723] disk space is not compliant W1122 04:48:57.450914 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:48:57.459715 3798194 disk_worker.go:509] system disk:vda1 I1122 04:48:57.459757 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:48:58.445993 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:48:59.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:48:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:48:59.450132 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:48:59.471672 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:49:02.108395 3798194 disk_info.go:125] begin check local disk info of client I1122 04:49:02.112295 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:49:02.112306 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003952c0 0xc000395300] E1122 04:49:06.413412 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:49:06.413439 3798194 memory.go:184] no items to output this cycle I1122 04:49:06.413596 3798194 cpu.go:275] no items to output this cycle E1122 04:49:16.411408 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:49:16.411429 3798194 memory.go:184] no items to output this cycle I1122 04:49:16.411576 3798194 cpu.go:275] no items to output this cycle E1122 04:49:26.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:49:26.412279 3798194 memory.go:191] Add success. I1122 04:49:26.416662 3798194 cpu.go:282] Add success. I1122 04:49:26.442215 3798194 net.go:642] Add success. I1122 04:49:26.443738 3798194 net.go:778] primary dev: ETH0 I1122 04:49:26.443752 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:49:26.443763 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:49:29.450567 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:49:29.450655 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:49:29.450680 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:49:36.412779 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:49:36.412800 3798194 memory.go:184] no items to output this cycle I1122 04:49:36.412947 3798194 cpu.go:275] no items to output this cycle E1122 04:49:46.412780 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:49:46.412802 3798194 memory.go:184] no items to output this cycle I1122 04:49:46.412946 3798194 cpu.go:275] no items to output this cycle E1122 04:49:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:49:56.411953 3798194 memory.go:191] Add success. W1122 04:49:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:49:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:49:56.412002 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:49:56.412348 3798194 cpu.go:282] Add success. I1122 04:49:56.441831 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:49:56.442429 3798194 net.go:642] Add success. I1122 04:49:56.446424 3798194 net.go:778] primary dev: ETH0 I1122 04:49:56.446441 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:49:56.446460 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:49:57.451290 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:49:57.451508 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:49:57.451520 3798194 disk_worker.go:723] disk space is not compliant W1122 04:49:57.451523 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:49:57.459910 3798194 disk_worker.go:509] system disk:vda1 I1122 04:49:57.459955 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:49:58.445997 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:49:59.450139 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:49:59.450219 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:49:59.450248 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:49:59.470729 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:50:02.112411 3798194 disk_info.go:125] begin check local disk info of client I1122 04:50:02.116294 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:50:02.116304 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004bafc0 0xc0004bb000] E1122 04:50:06.412844 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:50:06.412866 3798194 memory.go:184] no items to output this cycle I1122 04:50:06.413011 3798194 cpu.go:275] no items to output this cycle E1122 04:50:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:50:16.411978 3798194 memory.go:184] no items to output this cycle I1122 04:50:16.412134 3798194 cpu.go:275] no items to output this cycle E1122 04:50:26.411908 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:50:26.411946 3798194 memory.go:191] Add success. I1122 04:50:26.412296 3798194 cpu.go:282] Add success. I1122 04:50:26.441890 3798194 net.go:642] Add success. I1122 04:50:26.445764 3798194 net.go:778] primary dev: ETH0 I1122 04:50:26.445783 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:50:26.445796 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:50:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:50:29.450085 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:50:29.450120 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:50:36.412923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:50:36.412950 3798194 memory.go:184] no items to output this cycle I1122 04:50:36.413094 3798194 cpu.go:275] no items to output this cycle E1122 04:50:46.413107 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:50:46.413142 3798194 memory.go:184] no items to output this cycle I1122 04:50:46.413299 3798194 cpu.go:275] no items to output this cycle E1122 04:50:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:50:56.412267 3798194 memory.go:191] Add success. W1122 04:50:56.412307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:50:56.412319 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:50:56.412322 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:50:56.417600 3798194 cpu.go:282] Add success. I1122 04:50:56.444033 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:50:56.444630 3798194 net.go:642] Add success. I1122 04:50:56.446521 3798194 net.go:778] primary dev: ETH0 I1122 04:50:56.446539 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:50:56.446550 3798194 net.go:659] Add eni metrics success, count: 7 E1122 04:50:57.451254 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 04:50:57.451268 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:50:57.451469 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:50:57.451479 3798194 disk_worker.go:723] disk space is not compliant W1122 04:50:57.451482 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:50:57.459894 3798194 disk_worker.go:509] system disk:vda1 I1122 04:50:57.459935 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 04:50:58.448478 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 04:50:58.448493 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 04:50:59.448789 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 04:50:59.449863 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:50:59.449915 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:50:59.449933 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:50:59.471463 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:51:02.119136 3798194 disk_info.go:125] begin check local disk info of client I1122 04:51:02.122907 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:51:02.122916 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004321c0 0xc000432200] E1122 04:51:06.412348 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:51:06.412371 3798194 memory.go:184] no items to output this cycle I1122 04:51:06.412515 3798194 cpu.go:275] no items to output this cycle E1122 04:51:16.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:51:16.413377 3798194 memory.go:184] no items to output this cycle I1122 04:51:16.413665 3798194 cpu.go:275] no items to output this cycle I1122 04:51:17.632880 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:51:17.632888 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:51:26.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:51:26.412891 3798194 memory.go:191] Add success. I1122 04:51:26.417063 3798194 cpu.go:282] Add success. I1122 04:51:26.441610 3798194 net.go:642] Add success. I1122 04:51:26.443009 3798194 net.go:778] primary dev: ETH0 I1122 04:51:26.443026 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:51:26.443039 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:51:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:51:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:51:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:51:36.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:51:36.411987 3798194 memory.go:184] no items to output this cycle I1122 04:51:36.412139 3798194 cpu.go:275] no items to output this cycle E1122 04:51:46.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:51:46.412889 3798194 memory.go:184] no items to output this cycle I1122 04:51:46.413035 3798194 cpu.go:275] no items to output this cycle E1122 04:51:56.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:51:56.412883 3798194 memory.go:191] Add success. W1122 04:51:56.412911 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:51:56.412922 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:51:56.412926 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:51:56.417268 3798194 cpu.go:282] Add success. I1122 04:51:56.441672 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:51:56.442180 3798194 net.go:642] Add success. I1122 04:51:56.443646 3798194 net.go:778] primary dev: ETH0 I1122 04:51:56.443661 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:51:56.443673 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:51:57.450954 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:51:57.451174 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:51:57.451186 3798194 disk_worker.go:723] disk space is not compliant W1122 04:51:57.451190 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:51:57.459944 3798194 disk_worker.go:509] system disk:vda1 I1122 04:51:57.459985 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:51:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:51:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:51:59.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:51:59.450501 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:51:59.471010 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:52:02.125147 3798194 disk_info.go:125] begin check local disk info of client I1122 04:52:02.128940 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:52:02.128949 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037e100 0xc00037e140] E1122 04:52:06.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:52:06.413262 3798194 memory.go:184] no items to output this cycle I1122 04:52:06.413403 3798194 cpu.go:275] no items to output this cycle E1122 04:52:16.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:52:16.412886 3798194 memory.go:184] no items to output this cycle I1122 04:52:16.413032 3798194 cpu.go:275] no items to output this cycle E1122 04:52:26.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:52:26.412885 3798194 memory.go:191] Add success. I1122 04:52:26.417120 3798194 cpu.go:282] Add success. I1122 04:52:26.441644 3798194 net.go:642] Add success. I1122 04:52:26.443211 3798194 net.go:778] primary dev: ETH0 I1122 04:52:26.443226 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:52:26.443240 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:52:29.450438 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:52:29.450524 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:52:29.450550 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:52:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:52:36.412256 3798194 memory.go:184] no items to output this cycle I1122 04:52:36.412409 3798194 cpu.go:275] no items to output this cycle E1122 04:52:46.412075 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:52:46.412185 3798194 memory.go:184] no items to output this cycle I1122 04:52:46.412525 3798194 cpu.go:275] no items to output this cycle E1122 04:52:56.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:52:56.412277 3798194 memory.go:191] Add success. W1122 04:52:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:52:56.412323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:52:56.412326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:52:56.417067 3798194 cpu.go:282] Add success. I1122 04:52:56.441421 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:52:56.441886 3798194 net.go:642] Add success. I1122 04:52:56.443328 3798194 net.go:778] primary dev: ETH0 I1122 04:52:56.443343 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:52:56.443355 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:52:57.451009 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:52:57.451248 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:52:57.451259 3798194 disk_worker.go:723] disk space is not compliant W1122 04:52:57.451264 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:52:57.460149 3798194 disk_worker.go:509] system disk:vda1 I1122 04:52:57.460192 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:52:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:52:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:52:59.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:52:59.450504 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:52:59.471016 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:53:02.131142 3798194 disk_info.go:125] begin check local disk info of client I1122 04:53:02.134934 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:53:02.134942 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000410280 0xc0004102c0] E1122 04:53:06.412595 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:53:06.412617 3798194 memory.go:184] no items to output this cycle I1122 04:53:06.412757 3798194 cpu.go:275] no items to output this cycle E1122 04:53:16.413250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:53:16.413277 3798194 memory.go:184] no items to output this cycle I1122 04:53:16.413428 3798194 cpu.go:275] no items to output this cycle E1122 04:53:26.413250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:53:26.413285 3798194 memory.go:191] Add success. I1122 04:53:26.417912 3798194 cpu.go:282] Add success. I1122 04:53:26.441483 3798194 net.go:642] Add success. I1122 04:53:26.443101 3798194 net.go:778] primary dev: ETH0 I1122 04:53:26.443126 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:53:26.443139 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:53:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:53:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:53:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:53:36.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:53:36.412875 3798194 memory.go:184] no items to output this cycle I1122 04:53:36.413036 3798194 cpu.go:275] no items to output this cycle E1122 04:53:46.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:53:46.413271 3798194 memory.go:184] no items to output this cycle I1122 04:53:46.413427 3798194 cpu.go:275] no items to output this cycle E1122 04:53:56.413084 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:53:56.413141 3798194 memory.go:191] Add success. W1122 04:53:56.413176 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:53:56.413188 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:53:56.413192 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:53:56.418201 3798194 cpu.go:282] Add success. I1122 04:53:56.441645 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:53:56.442172 3798194 net.go:642] Add success. I1122 04:53:56.443649 3798194 net.go:778] primary dev: ETH0 I1122 04:53:56.443664 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:53:56.443678 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:53:57.446555 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:53:57.446835 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:53:57.446848 3798194 disk_worker.go:723] disk space is not compliant W1122 04:53:57.446852 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:53:57.453269 3798194 disk_worker.go:509] system disk:vda1 I1122 04:53:57.453313 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:53:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:53:59.449876 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:53:59.449958 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:53:59.449985 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:53:59.471545 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:54:02.135048 3798194 disk_info.go:125] begin check local disk info of client I1122 04:54:02.138826 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:54:02.138836 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000480b80 0xc000480bc0] E1122 04:54:06.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:54:06.412883 3798194 memory.go:184] no items to output this cycle I1122 04:54:06.413033 3798194 cpu.go:275] no items to output this cycle E1122 04:54:16.413256 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:54:16.413281 3798194 memory.go:184] no items to output this cycle I1122 04:54:16.413437 3798194 cpu.go:275] no items to output this cycle I1122 04:54:17.633212 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:54:17.633221 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:54:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:54:26.413266 3798194 memory.go:191] Add success. I1122 04:54:26.417438 3798194 cpu.go:282] Add success. I1122 04:54:26.441998 3798194 net.go:642] Add success. I1122 04:54:26.443431 3798194 net.go:778] primary dev: ETH0 I1122 04:54:26.443447 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:54:26.443460 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:54:29.450086 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:54:29.450198 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:54:29.450230 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:54:36.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:54:36.413272 3798194 memory.go:184] no items to output this cycle I1122 04:54:36.413412 3798194 cpu.go:275] no items to output this cycle E1122 04:54:46.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:54:46.413269 3798194 memory.go:184] no items to output this cycle I1122 04:54:46.413425 3798194 cpu.go:275] no items to output this cycle E1122 04:54:56.413017 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:54:56.413052 3798194 memory.go:191] Add success. W1122 04:54:56.413085 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:54:56.413097 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:54:56.413100 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:54:56.417556 3798194 cpu.go:282] Add success. I1122 04:54:56.440967 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:54:56.441561 3798194 net.go:642] Add success. I1122 04:54:56.443086 3798194 net.go:778] primary dev: ETH0 I1122 04:54:56.443101 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:54:56.443130 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:54:57.448522 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:54:57.448801 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:54:57.448814 3798194 disk_worker.go:723] disk space is not compliant W1122 04:54:57.448817 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:54:57.450916 3798194 disk_worker.go:509] system disk:vda1 I1122 04:54:57.450971 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:54:58.446416 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:54:59.450514 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:54:59.450593 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:54:59.450620 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:54:59.471127 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:55:02.142137 3798194 disk_info.go:125] begin check local disk info of client I1122 04:55:02.145914 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:55:02.145923 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00039fac0 0xc00039fb00] E1122 04:55:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:55:06.412259 3798194 memory.go:184] no items to output this cycle I1122 04:55:06.412410 3798194 cpu.go:275] no items to output this cycle E1122 04:55:16.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:55:16.413261 3798194 memory.go:184] no items to output this cycle I1122 04:55:16.413419 3798194 cpu.go:275] no items to output this cycle E1122 04:55:26.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:55:26.412274 3798194 memory.go:191] Add success. I1122 04:55:26.416669 3798194 cpu.go:282] Add success. I1122 04:55:26.441316 3798194 net.go:642] Add success. I1122 04:55:26.442804 3798194 net.go:778] primary dev: ETH0 I1122 04:55:26.442819 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:55:26.442831 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:55:29.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:55:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:55:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:55:36.412981 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:55:36.413004 3798194 memory.go:184] no items to output this cycle I1122 04:55:36.413181 3798194 cpu.go:275] no items to output this cycle E1122 04:55:46.412840 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:55:46.412862 3798194 memory.go:184] no items to output this cycle I1122 04:55:46.413010 3798194 cpu.go:275] no items to output this cycle E1122 04:55:56.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:55:56.413282 3798194 memory.go:191] Add success. W1122 04:55:56.413314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:55:56.413327 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:55:56.413330 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:55:56.417862 3798194 cpu.go:282] Add success. I1122 04:55:56.444311 3798194 event_worker.go:152] Polling the log file for events... I1122 04:55:56.444546 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:55:56.445021 3798194 net.go:642] Add success. I1122 04:55:56.446705 3798194 net.go:778] primary dev: ETH0 I1122 04:55:56.446720 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:55:56.446738 3798194 net.go:659] Add eni metrics success, count: 7 E1122 04:55:57.451009 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 04:55:57.451024 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:55:57.451249 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:55:57.451259 3798194 disk_worker.go:723] disk space is not compliant W1122 04:55:57.451262 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:55:57.460036 3798194 disk_worker.go:509] system disk:vda1 I1122 04:55:57.460080 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 04:55:58.448164 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 04:55:58.448180 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:55:59.449719 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:55:59.449778 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:55:59.449798 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:55:59.449924 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 04:55:59.471371 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:56:02.146024 3798194 disk_info.go:125] begin check local disk info of client I1122 04:56:02.149872 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:56:02.149884 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00027ff40 0xc000322000] E1122 04:56:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:56:06.412254 3798194 memory.go:184] no items to output this cycle I1122 04:56:06.412396 3798194 cpu.go:275] no items to output this cycle E1122 04:56:16.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:56:16.412880 3798194 memory.go:184] no items to output this cycle I1122 04:56:16.413023 3798194 cpu.go:275] no items to output this cycle E1122 04:56:26.413376 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:56:26.413411 3798194 memory.go:191] Add success. I1122 04:56:26.418025 3798194 cpu.go:282] Add success. I1122 04:56:26.441588 3798194 net.go:642] Add success. I1122 04:56:26.443261 3798194 net.go:778] primary dev: ETH0 I1122 04:56:26.443276 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:56:26.443291 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:56:29.450043 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:56:29.450139 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:56:29.450165 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:56:36.413206 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:56:36.413227 3798194 memory.go:184] no items to output this cycle I1122 04:56:36.413377 3798194 cpu.go:275] no items to output this cycle E1122 04:56:46.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:56:46.412255 3798194 memory.go:184] no items to output this cycle I1122 04:56:46.412400 3798194 cpu.go:275] no items to output this cycle E1122 04:56:56.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:56:56.412264 3798194 memory.go:191] Add success. W1122 04:56:56.412296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:56:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:56:56.412313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:56:56.416820 3798194 cpu.go:282] Add success. I1122 04:56:56.440927 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:56:56.441542 3798194 net.go:642] Add success. I1122 04:56:56.443083 3798194 net.go:778] primary dev: ETH0 I1122 04:56:56.443098 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:56:56.443127 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:56:57.448728 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:56:57.449006 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:56:57.449018 3798194 disk_worker.go:723] disk space is not compliant W1122 04:56:57.449021 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:56:57.452307 3798194 disk_worker.go:509] system disk:vda1 I1122 04:56:57.452345 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:56:58.446545 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:56:59.449666 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:56:59.449751 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:56:59.449777 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:56:59.471339 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:57:02.152141 3798194 disk_info.go:125] begin check local disk info of client I1122 04:57:02.156013 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:57:02.156021 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030f7c0 0xc00030f800] E1122 04:57:06.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:57:06.413261 3798194 memory.go:184] no items to output this cycle I1122 04:57:06.413420 3798194 cpu.go:275] no items to output this cycle E1122 04:57:16.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:57:16.413262 3798194 memory.go:184] no items to output this cycle I1122 04:57:16.413409 3798194 cpu.go:275] no items to output this cycle I1122 04:57:17.633384 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 04:57:17.633393 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 04:57:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:57:26.413270 3798194 memory.go:191] Add success. I1122 04:57:26.418235 3798194 cpu.go:282] Add success. I1122 04:57:26.441744 3798194 net.go:642] Add success. I1122 04:57:26.443361 3798194 net.go:778] primary dev: ETH0 I1122 04:57:26.443379 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:57:26.443391 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:57:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:57:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:57:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:57:36.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:57:36.412873 3798194 memory.go:184] no items to output this cycle I1122 04:57:36.413013 3798194 cpu.go:275] no items to output this cycle E1122 04:57:46.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:57:46.413261 3798194 memory.go:184] no items to output this cycle I1122 04:57:46.413405 3798194 cpu.go:275] no items to output this cycle E1122 04:57:56.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:57:56.413269 3798194 memory.go:191] Add success. W1122 04:57:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:57:56.413325 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:57:56.413331 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:57:56.417882 3798194 cpu.go:282] Add success. I1122 04:57:56.441257 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:57:56.441749 3798194 net.go:642] Add success. I1122 04:57:56.443386 3798194 net.go:778] primary dev: ETH0 I1122 04:57:56.443400 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:57:56.443414 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:57:57.446272 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:57:57.446541 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:57:57.446553 3798194 disk_worker.go:723] disk space is not compliant W1122 04:57:57.446556 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:57:57.451623 3798194 disk_worker.go:509] system disk:vda1 I1122 04:57:57.451663 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:57:58.445983 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:57:59.450496 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:57:59.450582 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:57:59.450607 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:57:59.471137 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:58:02.158139 3798194 disk_info.go:125] begin check local disk info of client I1122 04:58:02.161954 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:58:02.161973 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040fbc0 0xc00040fc00] E1122 04:58:06.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:58:06.412886 3798194 memory.go:184] no items to output this cycle I1122 04:58:06.413232 3798194 cpu.go:275] no items to output this cycle E1122 04:58:16.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:58:16.412262 3798194 memory.go:184] no items to output this cycle I1122 04:58:16.412408 3798194 cpu.go:275] no items to output this cycle E1122 04:58:26.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:58:26.412268 3798194 memory.go:191] Add success. I1122 04:58:26.417043 3798194 cpu.go:282] Add success. I1122 04:58:26.441617 3798194 net.go:642] Add success. I1122 04:58:26.443188 3798194 net.go:778] primary dev: ETH0 I1122 04:58:26.443202 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:58:26.443217 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:58:29.449974 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:58:29.450052 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:58:29.450083 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:58:36.412931 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:58:36.412955 3798194 memory.go:184] no items to output this cycle I1122 04:58:36.413094 3798194 cpu.go:275] no items to output this cycle E1122 04:58:46.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:58:46.412884 3798194 memory.go:184] no items to output this cycle I1122 04:58:46.413034 3798194 cpu.go:275] no items to output this cycle E1122 04:58:56.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:58:56.413273 3798194 memory.go:191] Add success. W1122 04:58:56.413307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:58:56.413320 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:58:56.413323 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:58:56.417828 3798194 cpu.go:282] Add success. I1122 04:58:56.441277 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:58:56.441823 3798194 net.go:642] Add success. I1122 04:58:56.443597 3798194 net.go:778] primary dev: ETH0 I1122 04:58:56.443613 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:58:56.443626 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:58:57.454035 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:58:57.454222 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:58:57.454279 3798194 disk_worker.go:723] disk space is not compliant W1122 04:58:57.454282 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:58:57.463443 3798194 disk_worker.go:509] system disk:vda1 I1122 04:58:57.463485 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:58:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:58:59.450107 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:58:59.450195 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:58:59.450221 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:58:59.470696 3798194 disk_local_worker.go:436] Get disk info: [] I1122 04:59:02.162076 3798194 disk_info.go:125] begin check local disk info of client I1122 04:59:02.165951 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 04:59:02.165962 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000370f80 0xc000370fc0] E1122 04:59:06.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:59:06.413264 3798194 memory.go:184] no items to output this cycle I1122 04:59:06.413404 3798194 cpu.go:275] no items to output this cycle E1122 04:59:16.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:59:16.412260 3798194 memory.go:184] no items to output this cycle I1122 04:59:16.412411 3798194 cpu.go:275] no items to output this cycle E1122 04:59:26.413251 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:59:26.413287 3798194 memory.go:191] Add success. I1122 04:59:26.417669 3798194 cpu.go:282] Add success. I1122 04:59:26.442152 3798194 net.go:642] Add success. I1122 04:59:26.443536 3798194 net.go:778] primary dev: ETH0 I1122 04:59:26.443551 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:59:26.443566 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:59:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:59:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:59:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 04:59:36.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:59:36.412250 3798194 memory.go:184] no items to output this cycle I1122 04:59:36.412401 3798194 cpu.go:275] no items to output this cycle E1122 04:59:46.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:59:46.413272 3798194 memory.go:184] no items to output this cycle I1122 04:59:46.413421 3798194 cpu.go:275] no items to output this cycle E1122 04:59:56.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 04:59:56.413283 3798194 memory.go:191] Add success. W1122 04:59:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 04:59:56.413323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 04:59:56.413326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 04:59:56.417510 3798194 cpu.go:282] Add success. I1122 04:59:56.441896 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 04:59:56.442431 3798194 net.go:642] Add success. I1122 04:59:56.443999 3798194 net.go:778] primary dev: ETH0 I1122 04:59:56.444014 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 04:59:56.444027 3798194 net.go:659] Add eni metrics success, count: 7 I1122 04:59:57.450585 3798194 custom_config.go:106] query custom config with name: gpu W1122 04:59:57.450745 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 04:59:57.450797 3798194 disk_worker.go:723] disk space is not compliant W1122 04:59:57.450800 3798194 disk_worker.go:743] disk inode is not compliant I1122 04:59:57.459091 3798194 disk_worker.go:509] system disk:vda1 I1122 04:59:57.459158 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 04:59:58.446613 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 04:59:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 04:59:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 04:59:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 04:59:59.471654 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:00:02.168143 3798194 disk_info.go:125] begin check local disk info of client I1122 05:00:02.171977 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:00:02.171986 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e5180 0xc0003e51c0] E1122 05:00:06.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:00:06.413262 3798194 memory.go:184] no items to output this cycle I1122 05:00:06.413409 3798194 cpu.go:275] no items to output this cycle E1122 05:00:16.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:00:16.413260 3798194 memory.go:184] no items to output this cycle I1122 05:00:16.413403 3798194 cpu.go:275] no items to output this cycle I1122 05:00:17.633550 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:00:17.633559 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:00:26.411079 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:00:26.411131 3798194 memory.go:191] Add success. I1122 05:00:26.416165 3798194 cpu.go:282] Add success. I1122 05:00:26.441691 3798194 net.go:642] Add success. I1122 05:00:26.443398 3798194 net.go:778] primary dev: ETH0 I1122 05:00:26.443414 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:00:26.443430 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:00:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:00:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:00:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:00:36.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:00:36.412246 3798194 memory.go:184] no items to output this cycle I1122 05:00:36.412388 3798194 cpu.go:275] no items to output this cycle E1122 05:00:46.412250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:00:46.412272 3798194 memory.go:184] no items to output this cycle I1122 05:00:46.412421 3798194 cpu.go:275] no items to output this cycle E1122 05:00:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:00:56.412264 3798194 memory.go:191] Add success. W1122 05:00:56.412290 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:00:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:00:56.412305 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:00:56.416662 3798194 cpu.go:282] Add success. I1122 05:00:56.443806 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:00:56.444304 3798194 net.go:642] Add success. I1122 05:00:56.445802 3798194 net.go:778] primary dev: ETH0 I1122 05:00:56.445816 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:00:56.445828 3798194 net.go:659] Add eni metrics success, count: 7 E1122 05:00:57.450715 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 05:00:57.450728 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:00:57.450879 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:00:57.451036 3798194 disk_worker.go:723] disk space is not compliant W1122 05:00:57.451041 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:00:57.460259 3798194 disk_worker.go:509] system disk:vda1 I1122 05:00:57.460301 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 05:00:58.448144 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 05:00:58.448159 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 05:00:59.448633 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 05:00:59.449712 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:00:59.449766 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:00:59.449786 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:00:59.471321 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:01:02.174144 3798194 disk_info.go:125] begin check local disk info of client I1122 05:01:02.177942 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:01:02.177950 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000487900 0xc000487940] E1122 05:01:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:01:06.412254 3798194 memory.go:184] no items to output this cycle I1122 05:01:06.412397 3798194 cpu.go:275] no items to output this cycle E1122 05:01:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:01:16.413258 3798194 memory.go:184] no items to output this cycle I1122 05:01:16.413414 3798194 cpu.go:275] no items to output this cycle E1122 05:01:26.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:01:26.413264 3798194 memory.go:191] Add success. I1122 05:01:26.417478 3798194 cpu.go:282] Add success. I1122 05:01:26.442024 3798194 net.go:642] Add success. I1122 05:01:26.443436 3798194 net.go:778] primary dev: ETH0 I1122 05:01:26.443452 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:01:26.443467 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:01:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:01:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:01:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:01:36.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:01:36.412267 3798194 memory.go:184] no items to output this cycle I1122 05:01:36.412411 3798194 cpu.go:275] no items to output this cycle E1122 05:01:46.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:01:46.412255 3798194 memory.go:184] no items to output this cycle I1122 05:01:46.412403 3798194 cpu.go:275] no items to output this cycle E1122 05:01:56.413222 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:01:56.413253 3798194 memory.go:191] Add success. W1122 05:01:56.413280 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:01:56.413291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:01:56.413295 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:01:56.417541 3798194 cpu.go:282] Add success. I1122 05:01:56.441946 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:01:56.442531 3798194 net.go:642] Add success. I1122 05:01:56.443914 3798194 net.go:778] primary dev: ETH0 I1122 05:01:56.443929 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:01:56.443941 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:01:57.451249 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:01:57.451450 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:01:57.451460 3798194 disk_worker.go:723] disk space is not compliant W1122 05:01:57.451463 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:01:57.459934 3798194 disk_worker.go:509] system disk:vda1 I1122 05:01:57.459992 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:01:58.445690 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:01:59.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:01:59.450467 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:01:59.450492 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:01:59.471001 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:02:02.180995 3798194 disk_info.go:125] begin check local disk info of client I1122 05:02:02.184829 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:02:02.184837 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003eed00 0xc0003eed40] E1122 05:02:06.412292 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:02:06.412315 3798194 memory.go:184] no items to output this cycle I1122 05:02:06.412463 3798194 cpu.go:275] no items to output this cycle E1122 05:02:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:02:16.413263 3798194 memory.go:184] no items to output this cycle I1122 05:02:16.413412 3798194 cpu.go:275] no items to output this cycle E1122 05:02:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:02:26.413275 3798194 memory.go:191] Add success. I1122 05:02:26.417862 3798194 cpu.go:282] Add success. I1122 05:02:26.442154 3798194 net.go:642] Add success. I1122 05:02:26.443812 3798194 net.go:778] primary dev: ETH0 I1122 05:02:26.443827 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:02:26.443840 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:02:29.449932 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:02:29.450017 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:02:29.450047 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:02:36.412671 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:02:36.412693 3798194 memory.go:184] no items to output this cycle I1122 05:02:36.412838 3798194 cpu.go:275] no items to output this cycle E1122 05:02:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:02:46.411975 3798194 memory.go:184] no items to output this cycle I1122 05:02:46.412143 3798194 cpu.go:275] no items to output this cycle E1122 05:02:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:02:56.411951 3798194 memory.go:191] Add success. W1122 05:02:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:02:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:02:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:02:56.412345 3798194 cpu.go:282] Add success. I1122 05:02:56.441845 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:02:56.442342 3798194 net.go:642] Add success. I1122 05:02:56.446880 3798194 net.go:778] primary dev: ETH0 I1122 05:02:56.446895 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:02:56.446909 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:02:57.450660 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:02:57.450869 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:02:57.450881 3798194 disk_worker.go:723] disk space is not compliant W1122 05:02:57.450884 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:02:57.459780 3798194 disk_worker.go:509] system disk:vda1 I1122 05:02:57.459822 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:02:58.445993 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:02:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:02:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:02:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:02:59.471697 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:03:02.188153 3798194 disk_info.go:125] begin check local disk info of client I1122 05:03:02.191921 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:03:02.191930 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e8840 0xc0002e8880] E1122 05:03:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:03:06.412251 3798194 memory.go:184] no items to output this cycle I1122 05:03:06.412393 3798194 cpu.go:275] no items to output this cycle E1122 05:03:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:03:16.411980 3798194 memory.go:184] no items to output this cycle I1122 05:03:16.412144 3798194 cpu.go:275] no items to output this cycle I1122 05:03:17.633729 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:03:17.633738 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:03:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:03:26.413266 3798194 memory.go:191] Add success. I1122 05:03:26.417798 3798194 cpu.go:282] Add success. I1122 05:03:26.441278 3798194 net.go:642] Add success. I1122 05:03:26.442896 3798194 net.go:778] primary dev: ETH0 I1122 05:03:26.442911 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:03:26.442925 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:03:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:03:29.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:03:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:03:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:03:36.412255 3798194 memory.go:184] no items to output this cycle I1122 05:03:36.412394 3798194 cpu.go:275] no items to output this cycle E1122 05:03:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:03:46.413257 3798194 memory.go:184] no items to output this cycle I1122 05:03:46.413404 3798194 cpu.go:275] no items to output this cycle E1122 05:03:56.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:03:56.412262 3798194 memory.go:191] Add success. W1122 05:03:56.412289 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:03:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:03:56.412304 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:03:56.416824 3798194 cpu.go:282] Add success. I1122 05:03:56.440975 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:03:56.441585 3798194 net.go:642] Add success. I1122 05:03:56.443174 3798194 net.go:778] primary dev: ETH0 I1122 05:03:56.443188 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:03:56.443201 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:03:57.450672 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:03:57.450884 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:03:57.450896 3798194 disk_worker.go:723] disk space is not compliant W1122 05:03:57.450899 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:03:57.459243 3798194 disk_worker.go:509] system disk:vda1 I1122 05:03:57.459285 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:03:58.445996 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:03:59.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:03:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:03:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:03:59.471648 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:04:02.195686 3798194 disk_info.go:125] begin check local disk info of client I1122 05:04:02.199525 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:04:02.199536 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e000 0xc00030e040] E1122 05:04:06.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:04:06.412876 3798194 memory.go:184] no items to output this cycle I1122 05:04:06.413018 3798194 cpu.go:275] no items to output this cycle E1122 05:04:16.412945 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:04:16.412969 3798194 memory.go:184] no items to output this cycle I1122 05:04:16.413138 3798194 cpu.go:275] no items to output this cycle E1122 05:04:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:04:26.413262 3798194 memory.go:191] Add success. I1122 05:04:26.417770 3798194 cpu.go:282] Add success. I1122 05:04:26.441286 3798194 net.go:642] Add success. I1122 05:04:26.442819 3798194 net.go:778] primary dev: ETH0 I1122 05:04:26.442834 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:04:26.442846 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:04:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:04:29.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:04:29.450129 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:04:36.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:04:36.412877 3798194 memory.go:184] no items to output this cycle I1122 05:04:36.413023 3798194 cpu.go:275] no items to output this cycle E1122 05:04:46.413091 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:04:46.413129 3798194 memory.go:184] no items to output this cycle I1122 05:04:46.413278 3798194 cpu.go:275] no items to output this cycle E1122 05:04:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:04:56.411954 3798194 memory.go:191] Add success. W1122 05:04:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:04:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:04:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:04:56.412361 3798194 cpu.go:282] Add success. I1122 05:04:56.441817 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:04:56.442323 3798194 net.go:642] Add success. I1122 05:04:56.446437 3798194 net.go:778] primary dev: ETH0 I1122 05:04:56.446452 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:04:56.446464 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:04:57.451512 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:04:57.451731 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:04:57.451742 3798194 disk_worker.go:723] disk space is not compliant W1122 05:04:57.451745 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:04:57.460437 3798194 disk_worker.go:509] system disk:vda1 I1122 05:04:57.460482 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:04:58.446381 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:04:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:04:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:04:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:04:59.471673 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:05:02.202138 3798194 disk_info.go:125] begin check local disk info of client I1122 05:05:02.205953 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:05:02.205962 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000450d40 0xc000450d80] E1122 05:05:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:05:06.411978 3798194 memory.go:184] no items to output this cycle I1122 05:05:06.412128 3798194 cpu.go:275] no items to output this cycle E1122 05:05:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:05:16.411983 3798194 memory.go:184] no items to output this cycle I1122 05:05:16.412151 3798194 cpu.go:275] no items to output this cycle E1122 05:05:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:05:26.411957 3798194 memory.go:191] Add success. I1122 05:05:26.412334 3798194 cpu.go:282] Add success. I1122 05:05:26.441936 3798194 net.go:642] Add success. I1122 05:05:26.445917 3798194 net.go:778] primary dev: ETH0 I1122 05:05:26.445931 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:05:26.445944 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:05:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:05:29.450127 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:05:29.450156 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:05:36.412759 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:05:36.412782 3798194 memory.go:184] no items to output this cycle I1122 05:05:36.412936 3798194 cpu.go:275] no items to output this cycle E1122 05:05:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:05:46.411977 3798194 memory.go:184] no items to output this cycle I1122 05:05:46.412130 3798194 cpu.go:275] no items to output this cycle E1122 05:05:56.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:05:56.412892 3798194 memory.go:191] Add success. W1122 05:05:56.412923 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:05:56.412935 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:05:56.412939 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:05:56.417096 3798194 cpu.go:282] Add success. I1122 05:05:56.441496 3798194 event_worker.go:152] Polling the log file for events... I1122 05:05:56.449383 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:05:56.449939 3798194 net.go:642] Add success. I1122 05:05:56.451485 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795293356, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 05:05:56.451609 3798194 net.go:778] primary dev: ETH0 I1122 05:05:56.451622 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:05:56.451636 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:05:56.462176 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 05:05:57.445588 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 05:05:57.445606 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:05:57.445812 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:05:57.445872 3798194 disk_worker.go:723] disk space is not compliant W1122 05:05:57.445876 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:05:57.450782 3798194 disk_worker.go:509] system disk:vda1 I1122 05:05:57.450823 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 05:05:58.448506 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 05:05:58.448528 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 05:05:59.448642 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 05:05:59.449719 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:05:59.449771 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:05:59.449795 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:05:59.471277 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:06:02.209145 3798194 disk_info.go:125] begin check local disk info of client I1122 05:06:02.213031 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:06:02.213040 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036a740 0xc00036a780] E1122 05:06:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:06:06.412257 3798194 memory.go:184] no items to output this cycle I1122 05:06:06.412401 3798194 cpu.go:275] no items to output this cycle E1122 05:06:16.412466 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:06:16.412489 3798194 memory.go:184] no items to output this cycle I1122 05:06:16.412646 3798194 cpu.go:275] no items to output this cycle I1122 05:06:17.633906 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:06:17.633914 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:06:26.411290 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:06:26.411324 3798194 memory.go:191] Add success. I1122 05:06:26.411650 3798194 cpu.go:282] Add success. I1122 05:06:26.441302 3798194 net.go:642] Add success. I1122 05:06:26.445406 3798194 net.go:778] primary dev: ETH0 I1122 05:06:26.445421 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:06:26.445433 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:06:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:06:29.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:06:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:06:36.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:06:36.411984 3798194 memory.go:184] no items to output this cycle I1122 05:06:36.412350 3798194 cpu.go:275] no items to output this cycle E1122 05:06:46.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:06:46.411988 3798194 memory.go:184] no items to output this cycle I1122 05:06:46.412148 3798194 cpu.go:275] no items to output this cycle E1122 05:06:56.411938 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:06:56.411972 3798194 memory.go:191] Add success. W1122 05:06:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:06:56.412016 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:06:56.412020 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:06:56.412418 3798194 cpu.go:282] Add success. I1122 05:06:56.441912 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:06:56.442495 3798194 net.go:642] Add success. I1122 05:06:56.447211 3798194 net.go:778] primary dev: ETH0 I1122 05:06:56.447225 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:06:56.447237 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:06:57.450750 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:06:57.450957 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:06:57.450969 3798194 disk_worker.go:723] disk space is not compliant W1122 05:06:57.450972 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:06:57.459735 3798194 disk_worker.go:509] system disk:vda1 I1122 05:06:57.459777 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:06:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:06:59.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:06:59.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:06:59.450500 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:06:59.471023 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:07:02.213153 3798194 disk_info.go:125] begin check local disk info of client I1122 05:07:02.217010 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:07:02.217036 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d0c00 0xc0003d0c40] E1122 05:07:06.413042 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:07:06.413066 3798194 memory.go:184] no items to output this cycle I1122 05:07:06.413229 3798194 cpu.go:275] no items to output this cycle E1122 05:07:16.413254 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:07:16.413277 3798194 memory.go:184] no items to output this cycle I1122 05:07:16.413427 3798194 cpu.go:275] no items to output this cycle E1122 05:07:26.412454 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:07:26.412487 3798194 memory.go:191] Add success. I1122 05:07:26.417533 3798194 cpu.go:282] Add success. I1122 05:07:26.442059 3798194 net.go:642] Add success. I1122 05:07:26.443591 3798194 net.go:778] primary dev: ETH0 I1122 05:07:26.443605 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:07:26.443619 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:07:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:07:29.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:07:29.450147 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:07:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:07:36.411979 3798194 memory.go:184] no items to output this cycle I1122 05:07:36.412132 3798194 cpu.go:275] no items to output this cycle E1122 05:07:46.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:07:46.412888 3798194 memory.go:184] no items to output this cycle I1122 05:07:46.413046 3798194 cpu.go:275] no items to output this cycle E1122 05:07:56.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:07:56.412265 3798194 memory.go:191] Add success. W1122 05:07:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:07:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:07:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:07:56.416533 3798194 cpu.go:282] Add success. I1122 05:07:56.441938 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:07:56.442529 3798194 net.go:642] Add success. I1122 05:07:56.444012 3798194 net.go:778] primary dev: ETH0 I1122 05:07:56.444026 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:07:56.444037 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:07:57.452155 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:07:57.452329 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:07:57.452386 3798194 disk_worker.go:723] disk space is not compliant W1122 05:07:57.452390 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:07:57.464315 3798194 disk_worker.go:509] system disk:vda1 I1122 05:07:57.464365 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:07:58.446008 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:07:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:07:59.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:07:59.450153 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:07:59.471706 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:08:02.217153 3798194 disk_info.go:125] begin check local disk info of client I1122 05:08:02.221061 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:08:02.221071 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000070000 0xc000070700] E1122 05:08:06.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:08:06.412262 3798194 memory.go:184] no items to output this cycle I1122 05:08:06.412409 3798194 cpu.go:275] no items to output this cycle E1122 05:08:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:08:16.412259 3798194 memory.go:184] no items to output this cycle I1122 05:08:16.412412 3798194 cpu.go:275] no items to output this cycle E1122 05:08:26.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:08:26.412276 3798194 memory.go:191] Add success. I1122 05:08:26.416738 3798194 cpu.go:282] Add success. I1122 05:08:26.441287 3798194 net.go:642] Add success. I1122 05:08:26.442813 3798194 net.go:778] primary dev: ETH0 I1122 05:08:26.442838 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:08:26.442852 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:08:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:08:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:08:29.450150 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:08:36.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:08:36.413264 3798194 memory.go:184] no items to output this cycle I1122 05:08:36.413405 3798194 cpu.go:275] no items to output this cycle E1122 05:08:46.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:08:46.413259 3798194 memory.go:184] no items to output this cycle I1122 05:08:46.413408 3798194 cpu.go:275] no items to output this cycle E1122 05:08:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:08:56.411949 3798194 memory.go:191] Add success. W1122 05:08:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:08:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:08:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:08:56.412354 3798194 cpu.go:282] Add success. I1122 05:08:56.441819 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:08:56.442423 3798194 net.go:642] Add success. I1122 05:08:56.446722 3798194 net.go:778] primary dev: ETH0 I1122 05:08:56.446744 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:08:56.446756 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:08:57.451311 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:08:57.451509 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:08:57.451520 3798194 disk_worker.go:723] disk space is not compliant W1122 05:08:57.451523 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:08:57.459956 3798194 disk_worker.go:509] system disk:vda1 I1122 05:08:57.460013 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:08:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:08:59.450386 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:08:59.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:08:59.450500 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:08:59.471034 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:09:02.224138 3798194 disk_info.go:125] begin check local disk info of client I1122 05:09:02.227980 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:09:02.227988 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040f240 0xc00040f280] E1122 05:09:06.413153 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:09:06.413175 3798194 memory.go:184] no items to output this cycle I1122 05:09:06.413324 3798194 cpu.go:275] no items to output this cycle E1122 05:09:16.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:09:16.411991 3798194 memory.go:184] no items to output this cycle I1122 05:09:16.412160 3798194 cpu.go:275] no items to output this cycle I1122 05:09:17.634342 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:09:17.634350 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:09:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:09:26.413269 3798194 memory.go:191] Add success. I1122 05:09:26.417767 3798194 cpu.go:282] Add success. I1122 05:09:26.441288 3798194 net.go:642] Add success. I1122 05:09:26.443058 3798194 net.go:778] primary dev: ETH0 I1122 05:09:26.443073 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:09:26.443086 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:09:29.449669 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:09:29.449758 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:09:29.449789 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:09:36.413349 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:09:36.413370 3798194 memory.go:184] no items to output this cycle I1122 05:09:36.413513 3798194 cpu.go:275] no items to output this cycle E1122 05:09:46.412745 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:09:46.412768 3798194 memory.go:184] no items to output this cycle I1122 05:09:46.412916 3798194 cpu.go:275] no items to output this cycle E1122 05:09:56.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:09:56.412259 3798194 memory.go:191] Add success. W1122 05:09:56.412292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:09:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:09:56.412308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:09:56.416721 3798194 cpu.go:282] Add success. I1122 05:09:56.441124 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:09:56.441666 3798194 net.go:642] Add success. I1122 05:09:56.443219 3798194 net.go:778] primary dev: ETH0 I1122 05:09:56.443234 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:09:56.443246 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:09:57.451060 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:09:57.451306 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:09:57.451318 3798194 disk_worker.go:723] disk space is not compliant W1122 05:09:57.451321 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:09:57.459788 3798194 disk_worker.go:509] system disk:vda1 I1122 05:09:57.459831 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:09:58.446381 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:09:59.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:09:59.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:09:59.450494 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:09:59.470956 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:10:02.228095 3798194 disk_info.go:125] begin check local disk info of client I1122 05:10:02.231976 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:10:02.231987 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030b400 0xc00030b440] E1122 05:10:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:10:06.411987 3798194 memory.go:184] no items to output this cycle I1122 05:10:06.412146 3798194 cpu.go:275] no items to output this cycle E1122 05:10:16.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:10:16.413263 3798194 memory.go:184] no items to output this cycle I1122 05:10:16.413407 3798194 cpu.go:275] no items to output this cycle E1122 05:10:26.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:10:26.412269 3798194 memory.go:191] Add success. I1122 05:10:26.416764 3798194 cpu.go:282] Add success. I1122 05:10:26.441290 3798194 net.go:642] Add success. I1122 05:10:26.442712 3798194 net.go:778] primary dev: ETH0 I1122 05:10:26.442728 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:10:26.442741 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:10:29.450384 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:10:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:10:29.450496 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:10:36.411988 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:10:36.412010 3798194 memory.go:184] no items to output this cycle I1122 05:10:36.412168 3798194 cpu.go:275] no items to output this cycle E1122 05:10:46.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:10:46.411984 3798194 memory.go:184] no items to output this cycle I1122 05:10:46.412146 3798194 cpu.go:275] no items to output this cycle E1122 05:10:56.412910 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:10:56.412947 3798194 memory.go:191] Add success. W1122 05:10:56.412979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:10:56.412990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:10:56.412994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:10:56.417462 3798194 cpu.go:282] Add success. I1122 05:10:56.444743 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:10:56.445330 3798194 net.go:642] Add success. I1122 05:10:56.446893 3798194 net.go:778] primary dev: ETH0 I1122 05:10:56.446907 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:10:56.446919 3798194 net.go:659] Add eni metrics success, count: 7 E1122 05:10:57.450686 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 05:10:57.450700 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:10:57.450863 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:10:57.450916 3798194 disk_worker.go:723] disk space is not compliant W1122 05:10:57.450919 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:10:57.459300 3798194 disk_worker.go:509] system disk:vda1 I1122 05:10:57.459342 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 05:10:58.448102 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 05:10:58.448131 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 05:10:59.448632 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 05:10:59.449707 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:10:59.449761 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:10:59.449784 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:10:59.471317 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:11:02.235213 3798194 disk_info.go:125] begin check local disk info of client I1122 05:11:02.239006 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:11:02.239014 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0ec0 0xc0002e0f00] E1122 05:11:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:11:06.411983 3798194 memory.go:184] no items to output this cycle I1122 05:11:06.412138 3798194 cpu.go:275] no items to output this cycle E1122 05:11:16.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:11:16.411991 3798194 memory.go:184] no items to output this cycle I1122 05:11:16.412155 3798194 cpu.go:275] no items to output this cycle E1122 05:11:26.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:11:26.412891 3798194 memory.go:191] Add success. I1122 05:11:26.417035 3798194 cpu.go:282] Add success. I1122 05:11:26.441573 3798194 net.go:642] Add success. I1122 05:11:26.443033 3798194 net.go:778] primary dev: ETH0 I1122 05:11:26.443047 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:11:26.443060 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:11:29.449660 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:11:29.449753 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:11:29.449783 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:11:36.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:11:36.412891 3798194 memory.go:184] no items to output this cycle I1122 05:11:36.413036 3798194 cpu.go:275] no items to output this cycle E1122 05:11:46.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:11:46.412250 3798194 memory.go:184] no items to output this cycle I1122 05:11:46.412396 3798194 cpu.go:275] no items to output this cycle E1122 05:11:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:11:56.411946 3798194 memory.go:191] Add success. W1122 05:11:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:11:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:11:56.411993 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:11:56.412340 3798194 cpu.go:282] Add success. I1122 05:11:56.441892 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:11:56.442400 3798194 net.go:642] Add success. I1122 05:11:56.446804 3798194 net.go:778] primary dev: ETH0 I1122 05:11:56.446819 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:11:56.446832 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:11:57.450633 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:11:57.450800 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:11:57.450855 3798194 disk_worker.go:723] disk space is not compliant W1122 05:11:57.450859 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:11:57.459231 3798194 disk_worker.go:509] system disk:vda1 I1122 05:11:57.459276 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:11:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:11:59.450476 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:11:59.450551 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:11:59.450577 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:11:59.471097 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:12:02.242922 3798194 disk_info.go:125] begin check local disk info of client I1122 05:12:02.246731 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:12:02.246741 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000482940 0xc000482980] E1122 05:12:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:12:06.411982 3798194 memory.go:184] no items to output this cycle I1122 05:12:06.412129 3798194 cpu.go:275] no items to output this cycle E1122 05:12:16.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:12:16.412883 3798194 memory.go:184] no items to output this cycle I1122 05:12:16.413028 3798194 cpu.go:275] no items to output this cycle I1122 05:12:17.634513 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:12:17.634521 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:12:26.412430 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:12:26.412462 3798194 memory.go:191] Add success. I1122 05:12:26.416779 3798194 cpu.go:282] Add success. I1122 05:12:26.441280 3798194 net.go:642] Add success. I1122 05:12:26.442845 3798194 net.go:778] primary dev: ETH0 I1122 05:12:26.442860 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:12:26.442874 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:12:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:12:29.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:12:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:12:36.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:12:36.412256 3798194 memory.go:184] no items to output this cycle I1122 05:12:36.412398 3798194 cpu.go:275] no items to output this cycle E1122 05:12:46.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:12:46.413253 3798194 memory.go:184] no items to output this cycle I1122 05:12:46.413402 3798194 cpu.go:275] no items to output this cycle E1122 05:12:56.412655 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:12:56.412689 3798194 memory.go:191] Add success. W1122 05:12:56.412718 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:12:56.412730 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:12:56.412734 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:12:56.416903 3798194 cpu.go:282] Add success. I1122 05:12:56.441432 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:12:56.441907 3798194 net.go:642] Add success. I1122 05:12:56.443559 3798194 net.go:778] primary dev: ETH0 I1122 05:12:56.443575 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:12:56.443587 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:12:57.450558 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:12:57.450753 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:12:57.450763 3798194 disk_worker.go:723] disk space is not compliant W1122 05:12:57.450766 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:12:57.459092 3798194 disk_worker.go:509] system disk:vda1 I1122 05:12:57.459147 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:12:58.446006 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:12:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:12:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:12:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:12:59.471663 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:13:02.250957 3798194 disk_info.go:125] begin check local disk info of client I1122 05:13:02.254747 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:13:02.254756 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003bc100 0xc0003bc140] E1122 05:13:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:13:06.411978 3798194 memory.go:184] no items to output this cycle I1122 05:13:06.412129 3798194 cpu.go:275] no items to output this cycle E1122 05:13:16.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:13:16.412255 3798194 memory.go:184] no items to output this cycle I1122 05:13:16.412400 3798194 cpu.go:275] no items to output this cycle E1122 05:13:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:13:26.411953 3798194 memory.go:191] Add success. I1122 05:13:26.412329 3798194 cpu.go:282] Add success. I1122 05:13:26.442023 3798194 net.go:642] Add success. I1122 05:13:26.446413 3798194 net.go:778] primary dev: ETH0 I1122 05:13:26.446432 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:13:26.446452 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:13:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:13:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:13:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:13:36.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:13:36.412249 3798194 memory.go:184] no items to output this cycle I1122 05:13:36.412390 3798194 cpu.go:275] no items to output this cycle E1122 05:13:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:13:46.413259 3798194 memory.go:184] no items to output this cycle I1122 05:13:46.413403 3798194 cpu.go:275] no items to output this cycle E1122 05:13:56.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:13:56.412889 3798194 memory.go:191] Add success. W1122 05:13:56.412915 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:13:56.412927 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:13:56.412930 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:13:56.417098 3798194 cpu.go:282] Add success. I1122 05:13:56.441595 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:13:56.442057 3798194 net.go:642] Add success. I1122 05:13:56.443569 3798194 net.go:778] primary dev: ETH0 I1122 05:13:56.443585 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:13:56.443598 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:13:57.451038 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:13:57.451257 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:13:57.451270 3798194 disk_worker.go:723] disk space is not compliant W1122 05:13:57.451274 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:13:57.460006 3798194 disk_worker.go:509] system disk:vda1 I1122 05:13:57.460046 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:13:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:13:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:13:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:13:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:13:59.471668 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:14:02.258145 3798194 disk_info.go:125] begin check local disk info of client I1122 05:14:02.261975 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:14:02.261984 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000468540 0xc000468580] E1122 05:14:06.412870 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:14:06.412891 3798194 memory.go:184] no items to output this cycle I1122 05:14:06.413039 3798194 cpu.go:275] no items to output this cycle E1122 05:14:16.412333 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:14:16.412356 3798194 memory.go:184] no items to output this cycle I1122 05:14:16.412502 3798194 cpu.go:275] no items to output this cycle E1122 05:14:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:14:26.411954 3798194 memory.go:191] Add success. I1122 05:14:26.412305 3798194 cpu.go:282] Add success. I1122 05:14:26.441907 3798194 net.go:642] Add success. I1122 05:14:26.446168 3798194 net.go:778] primary dev: ETH0 I1122 05:14:26.446183 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:14:26.446197 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:14:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:14:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:14:29.450501 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:14:36.412090 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:14:36.412127 3798194 memory.go:184] no items to output this cycle I1122 05:14:36.412274 3798194 cpu.go:275] no items to output this cycle E1122 05:14:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:14:46.411981 3798194 memory.go:184] no items to output this cycle I1122 05:14:46.412141 3798194 cpu.go:275] no items to output this cycle E1122 05:14:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:14:56.411955 3798194 memory.go:191] Add success. W1122 05:14:56.411981 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:14:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:14:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:14:56.412331 3798194 cpu.go:282] Add success. I1122 05:14:56.441793 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:14:56.442517 3798194 net.go:642] Add success. I1122 05:14:56.447582 3798194 net.go:778] primary dev: ETH0 I1122 05:14:56.447601 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:14:56.447616 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:14:57.450631 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:14:57.450843 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:14:57.450856 3798194 disk_worker.go:723] disk space is not compliant W1122 05:14:57.450859 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:14:57.459259 3798194 disk_worker.go:509] system disk:vda1 I1122 05:14:57.459299 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:14:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:14:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:14:59.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:14:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:14:59.471646 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:15:02.266032 3798194 disk_info.go:125] begin check local disk info of client I1122 05:15:02.269844 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:15:02.269853 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a39c0 0xc0004a3a00] E1122 05:15:06.413253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:15:06.413275 3798194 memory.go:184] no items to output this cycle I1122 05:15:06.413418 3798194 cpu.go:275] no items to output this cycle E1122 05:15:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:15:16.412260 3798194 memory.go:184] no items to output this cycle I1122 05:15:16.412408 3798194 cpu.go:275] no items to output this cycle I1122 05:15:17.634683 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:15:17.634691 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:15:26.412846 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:15:26.412880 3798194 memory.go:191] Add success. I1122 05:15:26.417128 3798194 cpu.go:282] Add success. I1122 05:15:26.441744 3798194 net.go:642] Add success. I1122 05:15:26.443161 3798194 net.go:778] primary dev: ETH0 I1122 05:15:26.443176 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:15:26.443191 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:15:29.449947 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:15:29.450032 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:15:29.450061 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:15:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:15:36.412257 3798194 memory.go:184] no items to output this cycle I1122 05:15:36.412410 3798194 cpu.go:275] no items to output this cycle E1122 05:15:46.412867 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:15:46.412889 3798194 memory.go:184] no items to output this cycle I1122 05:15:46.413035 3798194 cpu.go:275] no items to output this cycle E1122 05:15:56.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:15:56.413279 3798194 memory.go:191] Add success. W1122 05:15:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:15:56.413323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:15:56.413326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:15:56.417421 3798194 cpu.go:282] Add success. I1122 05:15:56.441694 3798194 event_worker.go:152] Polling the log file for events... I1122 05:15:56.444763 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:15:56.445286 3798194 net.go:642] Add success. I1122 05:15:56.446749 3798194 net.go:778] primary dev: ETH0 I1122 05:15:56.446765 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:15:56.446778 3798194 net.go:659] Add eni metrics success, count: 7 E1122 05:15:57.450632 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 05:15:57.450646 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:15:57.450848 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:15:57.450858 3798194 disk_worker.go:723] disk space is not compliant W1122 05:15:57.450861 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:15:57.459232 3798194 disk_worker.go:509] system disk:vda1 I1122 05:15:57.459277 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 05:15:58.447994 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 05:15:58.448008 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 05:15:59.448619 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 05:15:59.449692 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:15:59.449742 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:15:59.449763 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:15:59.471302 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:16:02.274070 3798194 disk_info.go:125] begin check local disk info of client I1122 05:16:02.277838 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:16:02.277847 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00043a180 0xc00043a1c0] E1122 05:16:06.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:16:06.411974 3798194 memory.go:184] no items to output this cycle I1122 05:16:06.412128 3798194 cpu.go:275] no items to output this cycle E1122 05:16:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:16:16.411974 3798194 memory.go:184] no items to output this cycle I1122 05:16:16.412241 3798194 cpu.go:275] no items to output this cycle E1122 05:16:26.411981 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:16:26.412014 3798194 memory.go:191] Add success. I1122 05:16:26.416283 3798194 cpu.go:282] Add success. I1122 05:16:26.441764 3798194 net.go:642] Add success. I1122 05:16:26.443245 3798194 net.go:778] primary dev: ETH0 I1122 05:16:26.443261 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:16:26.443274 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:16:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:16:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:16:29.450145 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:16:36.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:16:36.412249 3798194 memory.go:184] no items to output this cycle I1122 05:16:36.412395 3798194 cpu.go:275] no items to output this cycle E1122 05:16:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:16:46.411979 3798194 memory.go:184] no items to output this cycle I1122 05:16:46.412137 3798194 cpu.go:275] no items to output this cycle E1122 05:16:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:16:56.412267 3798194 memory.go:191] Add success. W1122 05:16:56.412298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:16:56.412310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:16:56.412313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:16:56.416726 3798194 cpu.go:282] Add success. I1122 05:16:56.441135 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:16:56.441605 3798194 net.go:642] Add success. I1122 05:16:56.443242 3798194 net.go:778] primary dev: ETH0 I1122 05:16:56.443259 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:16:56.443271 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:16:57.450629 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:16:57.450796 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:16:57.450854 3798194 disk_worker.go:723] disk space is not compliant W1122 05:16:57.450857 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:16:57.459154 3798194 disk_worker.go:509] system disk:vda1 I1122 05:16:57.459195 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:16:58.446372 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:16:59.450475 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:16:59.450558 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:16:59.450588 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:16:59.471105 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:17:02.281491 3798194 disk_info.go:125] begin check local disk info of client I1122 05:17:02.285263 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:17:02.285273 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00035d440 0xc00035d480] E1122 05:17:06.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:17:06.412881 3798194 memory.go:184] no items to output this cycle I1122 05:17:06.413021 3798194 cpu.go:275] no items to output this cycle E1122 05:17:16.412329 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:17:16.412351 3798194 memory.go:184] no items to output this cycle I1122 05:17:16.412506 3798194 cpu.go:275] no items to output this cycle E1122 05:17:26.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:17:26.412892 3798194 memory.go:191] Add success. I1122 05:17:26.417346 3798194 cpu.go:282] Add success. I1122 05:17:26.441893 3798194 net.go:642] Add success. I1122 05:17:26.443365 3798194 net.go:778] primary dev: ETH0 I1122 05:17:26.443381 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:17:26.443394 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:17:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:17:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:17:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:17:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:17:36.411977 3798194 memory.go:184] no items to output this cycle I1122 05:17:36.412130 3798194 cpu.go:275] no items to output this cycle E1122 05:17:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:17:46.411982 3798194 memory.go:184] no items to output this cycle I1122 05:17:46.412143 3798194 cpu.go:275] no items to output this cycle E1122 05:17:56.412433 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:17:56.412543 3798194 memory.go:191] Add success. W1122 05:17:56.412579 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:17:56.412591 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:17:56.412594 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:17:56.417264 3798194 cpu.go:282] Add success. I1122 05:17:56.440960 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:17:56.441475 3798194 net.go:642] Add success. I1122 05:17:56.442883 3798194 net.go:778] primary dev: ETH0 I1122 05:17:56.442898 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:17:56.442911 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:17:57.450746 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:17:57.450952 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:17:57.450963 3798194 disk_worker.go:723] disk space is not compliant W1122 05:17:57.450966 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:17:57.459796 3798194 disk_worker.go:509] system disk:vda1 I1122 05:17:57.459838 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:17:58.445994 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:17:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:17:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:17:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:17:59.471700 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:18:02.289149 3798194 disk_info.go:125] begin check local disk info of client I1122 05:18:02.292846 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:18:02.292856 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0dc0 0xc0002e0e00] E1122 05:18:06.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:18:06.411988 3798194 memory.go:184] no items to output this cycle I1122 05:18:06.412147 3798194 cpu.go:275] no items to output this cycle E1122 05:18:16.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:18:16.412252 3798194 memory.go:184] no items to output this cycle I1122 05:18:16.412400 3798194 cpu.go:275] no items to output this cycle I1122 05:18:17.635215 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:18:17.635223 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:18:26.411302 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:18:26.411337 3798194 memory.go:191] Add success. I1122 05:18:26.411647 3798194 cpu.go:282] Add success. I1122 05:18:26.441267 3798194 net.go:642] Add success. I1122 05:18:26.445359 3798194 net.go:778] primary dev: ETH0 I1122 05:18:26.445376 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:18:26.445389 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:18:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:18:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:18:29.450501 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:18:36.412821 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:18:36.412843 3798194 memory.go:184] no items to output this cycle I1122 05:18:36.412990 3798194 cpu.go:275] no items to output this cycle E1122 05:18:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:18:46.413260 3798194 memory.go:184] no items to output this cycle I1122 05:18:46.413413 3798194 cpu.go:275] no items to output this cycle E1122 05:18:56.411935 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:18:56.411969 3798194 memory.go:191] Add success. W1122 05:18:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:18:56.412014 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:18:56.412017 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:18:56.412365 3798194 cpu.go:282] Add success. I1122 05:18:56.441794 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:18:56.442375 3798194 net.go:642] Add success. I1122 05:18:56.448337 3798194 net.go:778] primary dev: ETH0 I1122 05:18:56.448353 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:18:56.448367 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:18:57.450646 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:18:57.450863 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:18:57.450874 3798194 disk_worker.go:723] disk space is not compliant W1122 05:18:57.450877 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:18:57.459279 3798194 disk_worker.go:509] system disk:vda1 I1122 05:18:57.459323 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:18:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:18:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:18:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:18:59.450150 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:18:59.471692 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:19:02.297138 3798194 disk_info.go:125] begin check local disk info of client I1122 05:19:02.300865 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:19:02.300873 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00047e180 0xc00047e1c0] E1122 05:19:06.411987 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:19:06.412090 3798194 memory.go:184] no items to output this cycle I1122 05:19:06.412568 3798194 cpu.go:275] no items to output this cycle E1122 05:19:16.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:19:16.413260 3798194 memory.go:184] no items to output this cycle I1122 05:19:16.413412 3798194 cpu.go:275] no items to output this cycle E1122 05:19:26.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:19:26.413270 3798194 memory.go:191] Add success. I1122 05:19:26.417964 3798194 cpu.go:282] Add success. I1122 05:19:26.441648 3798194 net.go:642] Add success. I1122 05:19:26.443198 3798194 net.go:778] primary dev: ETH0 I1122 05:19:26.443213 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:19:26.443228 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:19:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:19:29.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:19:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:19:36.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:19:36.412885 3798194 memory.go:184] no items to output this cycle I1122 05:19:36.413030 3798194 cpu.go:275] no items to output this cycle E1122 05:19:46.413323 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:19:46.413346 3798194 memory.go:184] no items to output this cycle I1122 05:19:46.413496 3798194 cpu.go:275] no items to output this cycle E1122 05:19:56.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:19:56.412893 3798194 memory.go:191] Add success. W1122 05:19:56.412927 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:19:56.412939 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:19:56.412942 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:19:56.417087 3798194 cpu.go:282] Add success. I1122 05:19:56.441430 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:19:56.441957 3798194 net.go:642] Add success. I1122 05:19:56.443525 3798194 net.go:778] primary dev: ETH0 I1122 05:19:56.443540 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:19:56.443553 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:19:57.450881 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:19:57.451087 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:19:57.451098 3798194 disk_worker.go:723] disk space is not compliant W1122 05:19:57.451101 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:19:57.459965 3798194 disk_worker.go:509] system disk:vda1 I1122 05:19:57.460010 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:19:58.446646 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:19:59.450006 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:19:59.450091 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:19:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:19:59.471646 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:20:02.305147 3798194 disk_info.go:125] begin check local disk info of client I1122 05:20:02.308864 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:20:02.308874 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000373380 0xc0003733c0] E1122 05:20:06.413457 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:20:06.413479 3798194 memory.go:184] no items to output this cycle I1122 05:20:06.413625 3798194 cpu.go:275] no items to output this cycle E1122 05:20:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:20:16.411973 3798194 memory.go:184] no items to output this cycle I1122 05:20:16.412140 3798194 cpu.go:275] no items to output this cycle E1122 05:20:26.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:20:26.412265 3798194 memory.go:191] Add success. I1122 05:20:26.416608 3798194 cpu.go:282] Add success. I1122 05:20:26.442259 3798194 net.go:642] Add success. I1122 05:20:26.443830 3798194 net.go:778] primary dev: ETH0 I1122 05:20:26.443845 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:20:26.443863 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:20:29.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:20:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:20:29.450144 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:20:36.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:20:36.413267 3798194 memory.go:184] no items to output this cycle I1122 05:20:36.413412 3798194 cpu.go:275] no items to output this cycle E1122 05:20:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:20:46.411978 3798194 memory.go:184] no items to output this cycle I1122 05:20:46.412136 3798194 cpu.go:275] no items to output this cycle E1122 05:20:56.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:20:56.413260 3798194 memory.go:191] Add success. W1122 05:20:56.413292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:20:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:20:56.413307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:20:56.417419 3798194 cpu.go:282] Add success. I1122 05:20:56.444700 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:20:56.445188 3798194 net.go:642] Add success. I1122 05:20:56.446661 3798194 net.go:778] primary dev: ETH0 I1122 05:20:56.446683 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:20:56.446696 3798194 net.go:659] Add eni metrics success, count: 7 E1122 05:20:57.450982 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 05:20:57.450995 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:20:57.451171 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:20:57.451229 3798194 disk_worker.go:723] disk space is not compliant W1122 05:20:57.451232 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:20:57.459989 3798194 disk_worker.go:509] system disk:vda1 I1122 05:20:57.460031 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 05:20:58.448448 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 05:20:58.448463 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 05:20:59.449138 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 05:20:59.450214 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:20:59.450269 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:20:59.450291 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:20:59.470831 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:21:02.312150 3798194 disk_info.go:125] begin check local disk info of client I1122 05:21:02.315900 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:21:02.315909 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c3940 0xc0003c3980] E1122 05:21:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:21:06.411984 3798194 memory.go:184] no items to output this cycle I1122 05:21:06.412143 3798194 cpu.go:275] no items to output this cycle E1122 05:21:16.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:21:16.413251 3798194 memory.go:184] no items to output this cycle I1122 05:21:16.413397 3798194 cpu.go:275] no items to output this cycle I1122 05:21:17.636213 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:21:17.636222 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:21:26.413443 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:21:26.413479 3798194 memory.go:191] Add success. I1122 05:21:26.417584 3798194 cpu.go:282] Add success. I1122 05:21:26.442134 3798194 net.go:642] Add success. I1122 05:21:26.443546 3798194 net.go:778] primary dev: ETH0 I1122 05:21:26.443561 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:21:26.443573 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:21:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:21:29.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:21:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:21:36.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:21:36.411989 3798194 memory.go:184] no items to output this cycle I1122 05:21:36.412143 3798194 cpu.go:275] no items to output this cycle E1122 05:21:46.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:21:46.412879 3798194 memory.go:184] no items to output this cycle I1122 05:21:46.413029 3798194 cpu.go:275] no items to output this cycle E1122 05:21:56.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:21:56.411954 3798194 memory.go:191] Add success. W1122 05:21:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:21:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:21:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:21:56.412519 3798194 cpu.go:282] Add success. I1122 05:21:56.440967 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:21:56.441568 3798194 net.go:642] Add success. I1122 05:21:56.445707 3798194 net.go:778] primary dev: ETH0 I1122 05:21:56.445722 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:21:56.445735 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:21:57.450691 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:21:57.450856 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:21:57.450911 3798194 disk_worker.go:723] disk space is not compliant W1122 05:21:57.450914 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:21:57.459207 3798194 disk_worker.go:509] system disk:vda1 I1122 05:21:57.459249 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:21:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:21:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:21:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:21:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:21:59.471640 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:22:02.320148 3798194 disk_info.go:125] begin check local disk info of client I1122 05:22:02.323907 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:22:02.323923 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e7c0 0xc00030e800] E1122 05:22:06.412842 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:22:06.412864 3798194 memory.go:184] no items to output this cycle I1122 05:22:06.413011 3798194 cpu.go:275] no items to output this cycle E1122 05:22:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:22:16.411978 3798194 memory.go:184] no items to output this cycle I1122 05:22:16.412138 3798194 cpu.go:275] no items to output this cycle E1122 05:22:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:22:26.411957 3798194 memory.go:191] Add success. I1122 05:22:26.412287 3798194 cpu.go:282] Add success. I1122 05:22:26.441748 3798194 net.go:642] Add success. I1122 05:22:26.445558 3798194 net.go:778] primary dev: ETH0 I1122 05:22:26.445574 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:22:26.445587 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:22:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:22:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:22:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:22:36.413167 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:22:36.413190 3798194 memory.go:184] no items to output this cycle I1122 05:22:36.413331 3798194 cpu.go:275] no items to output this cycle E1122 05:22:46.412870 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:22:46.412894 3798194 memory.go:184] no items to output this cycle I1122 05:22:46.413042 3798194 cpu.go:275] no items to output this cycle E1122 05:22:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:22:56.412265 3798194 memory.go:191] Add success. W1122 05:22:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:22:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:22:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:22:56.416906 3798194 cpu.go:282] Add success. I1122 05:22:56.441367 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:22:56.441942 3798194 net.go:642] Add success. I1122 05:22:56.443499 3798194 net.go:778] primary dev: ETH0 I1122 05:22:56.443516 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:22:56.443538 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:22:57.450986 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:22:57.451219 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:22:57.451232 3798194 disk_worker.go:723] disk space is not compliant W1122 05:22:57.451235 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:22:57.460028 3798194 disk_worker.go:509] system disk:vda1 I1122 05:22:57.460070 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:22:58.445997 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:22:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:22:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:22:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:22:59.471708 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:23:02.327143 3798194 disk_info.go:125] begin check local disk info of client I1122 05:23:02.330989 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:23:02.330997 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00042c440 0xc00042c480] E1122 05:23:06.412846 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:23:06.412868 3798194 memory.go:184] no items to output this cycle I1122 05:23:06.413022 3798194 cpu.go:275] no items to output this cycle E1122 05:23:16.412535 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:23:16.412557 3798194 memory.go:184] no items to output this cycle I1122 05:23:16.412703 3798194 cpu.go:275] no items to output this cycle E1122 05:23:26.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:23:26.411949 3798194 memory.go:191] Add success. I1122 05:23:26.412310 3798194 cpu.go:282] Add success. I1122 05:23:26.442020 3798194 net.go:642] Add success. I1122 05:23:26.446284 3798194 net.go:778] primary dev: ETH0 I1122 05:23:26.446300 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:23:26.446312 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:23:29.450073 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:23:29.450157 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:23:29.450186 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:23:36.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:23:36.413268 3798194 memory.go:184] no items to output this cycle I1122 05:23:36.413409 3798194 cpu.go:275] no items to output this cycle E1122 05:23:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:23:46.411974 3798194 memory.go:184] no items to output this cycle I1122 05:23:46.412136 3798194 cpu.go:275] no items to output this cycle E1122 05:23:56.413007 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:23:56.413041 3798194 memory.go:191] Add success. W1122 05:23:56.413070 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:23:56.413082 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:23:56.413085 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:23:56.417644 3798194 cpu.go:282] Add success. I1122 05:23:56.440966 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:23:56.441548 3798194 net.go:642] Add success. I1122 05:23:56.443057 3798194 net.go:778] primary dev: ETH0 I1122 05:23:56.443073 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:23:56.443085 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:23:57.450671 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:23:57.450834 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:23:57.450888 3798194 disk_worker.go:723] disk space is not compliant W1122 05:23:57.450891 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:23:57.459292 3798194 disk_worker.go:509] system disk:vda1 I1122 05:23:57.459332 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:23:58.446009 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:23:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:23:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:23:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:23:59.471657 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:24:02.335371 3798194 disk_info.go:125] begin check local disk info of client I1122 05:24:02.339243 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:24:02.339252 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a4180 0xc0003a41c0] E1122 05:24:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:24:06.412254 3798194 memory.go:184] no items to output this cycle I1122 05:24:06.412411 3798194 cpu.go:275] no items to output this cycle E1122 05:24:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:24:16.411982 3798194 memory.go:184] no items to output this cycle I1122 05:24:16.412148 3798194 cpu.go:275] no items to output this cycle I1122 05:24:17.637211 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:24:17.637219 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:24:26.411485 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:24:26.411516 3798194 memory.go:191] Add success. I1122 05:24:26.412003 3798194 cpu.go:282] Add success. I1122 05:24:26.441301 3798194 net.go:642] Add success. I1122 05:24:26.445624 3798194 net.go:778] primary dev: ETH0 I1122 05:24:26.445639 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:24:26.445651 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:24:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:24:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:24:29.450514 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:24:36.413101 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:24:36.413141 3798194 memory.go:184] no items to output this cycle I1122 05:24:36.413298 3798194 cpu.go:275] no items to output this cycle E1122 05:24:46.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:24:46.413255 3798194 memory.go:184] no items to output this cycle I1122 05:24:46.413398 3798194 cpu.go:275] no items to output this cycle E1122 05:24:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:24:56.411954 3798194 memory.go:191] Add success. W1122 05:24:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:24:56.412005 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:24:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:24:56.412368 3798194 cpu.go:282] Add success. I1122 05:24:56.441828 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:24:56.442407 3798194 net.go:642] Add success. I1122 05:24:56.446372 3798194 net.go:778] primary dev: ETH0 I1122 05:24:56.446388 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:24:56.446400 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:24:57.450711 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:24:57.450919 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:24:57.450930 3798194 disk_worker.go:723] disk space is not compliant W1122 05:24:57.450933 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:24:57.459749 3798194 disk_worker.go:509] system disk:vda1 I1122 05:24:57.459794 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:24:58.446014 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:24:59.450125 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:24:59.450209 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:24:59.450234 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:24:59.470722 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:25:02.343106 3798194 disk_info.go:125] begin check local disk info of client I1122 05:25:02.346938 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:25:02.346947 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000308840 0xc000308880] E1122 05:25:06.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:25:06.412250 3798194 memory.go:184] no items to output this cycle I1122 05:25:06.412398 3798194 cpu.go:275] no items to output this cycle E1122 05:25:16.412838 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:25:16.412862 3798194 memory.go:184] no items to output this cycle I1122 05:25:16.413020 3798194 cpu.go:275] no items to output this cycle E1122 05:25:26.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:25:26.411960 3798194 memory.go:191] Add success. I1122 05:25:26.412319 3798194 cpu.go:282] Add success. I1122 05:25:26.441925 3798194 net.go:642] Add success. I1122 05:25:26.446152 3798194 net.go:778] primary dev: ETH0 I1122 05:25:26.446169 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:25:26.446184 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:25:29.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:25:29.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:25:29.450148 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:25:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:25:36.412255 3798194 memory.go:184] no items to output this cycle I1122 05:25:36.412398 3798194 cpu.go:275] no items to output this cycle E1122 05:25:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:25:46.411975 3798194 memory.go:184] no items to output this cycle I1122 05:25:46.412143 3798194 cpu.go:275] no items to output this cycle E1122 05:25:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:25:56.411952 3798194 memory.go:191] Add success. W1122 05:25:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:25:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:25:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:25:56.412374 3798194 cpu.go:282] Add success. I1122 05:25:56.441578 3798194 event_worker.go:152] Polling the log file for events... I1122 05:25:56.444696 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:25:56.445258 3798194 net.go:642] Add success. I1122 05:25:56.449242 3798194 net.go:778] primary dev: ETH0 I1122 05:25:56.449258 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:25:56.449271 3798194 net.go:659] Add eni metrics success, count: 7 E1122 05:25:57.451083 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 05:25:57.451096 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:25:57.451312 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:25:57.451324 3798194 disk_worker.go:723] disk space is not compliant W1122 05:25:57.451327 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:25:57.459772 3798194 disk_worker.go:509] system disk:vda1 I1122 05:25:57.459814 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 05:25:58.448467 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 05:25:58.448481 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 05:25:59.448622 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 05:25:59.449697 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:25:59.449750 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:25:59.449769 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:25:59.471271 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:26:02.347048 3798194 disk_info.go:125] begin check local disk info of client I1122 05:26:02.350946 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:26:02.350955 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0340 0xc0002e0380] E1122 05:26:06.412995 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:26:06.413020 3798194 memory.go:184] no items to output this cycle I1122 05:26:06.413178 3798194 cpu.go:275] no items to output this cycle E1122 05:26:16.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:26:16.412260 3798194 memory.go:184] no items to output this cycle I1122 05:26:16.412440 3798194 cpu.go:275] no items to output this cycle E1122 05:26:26.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:26:26.413277 3798194 memory.go:191] Add success. I1122 05:26:26.417424 3798194 cpu.go:282] Add success. I1122 05:26:26.441968 3798194 net.go:642] Add success. I1122 05:26:26.443699 3798194 net.go:778] primary dev: ETH0 I1122 05:26:26.443715 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:26:26.443728 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:26:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:26:29.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:26:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:26:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:26:36.411979 3798194 memory.go:184] no items to output this cycle I1122 05:26:36.412149 3798194 cpu.go:275] no items to output this cycle E1122 05:26:46.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:26:46.413260 3798194 memory.go:184] no items to output this cycle I1122 05:26:46.413416 3798194 cpu.go:275] no items to output this cycle E1122 05:26:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:26:56.411956 3798194 memory.go:191] Add success. W1122 05:26:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:26:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:26:56.412007 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:26:56.412353 3798194 cpu.go:282] Add success. I1122 05:26:56.441812 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:26:56.442437 3798194 net.go:642] Add success. I1122 05:26:56.446476 3798194 net.go:778] primary dev: ETH0 I1122 05:26:56.446492 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:26:56.446505 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:26:57.450644 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:26:57.450855 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:26:57.450865 3798194 disk_worker.go:723] disk space is not compliant W1122 05:26:57.450868 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:26:57.460156 3798194 disk_worker.go:509] system disk:vda1 I1122 05:26:57.460198 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:26:58.446022 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:26:59.450127 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:26:59.450217 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:26:59.450244 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:26:59.470731 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:27:02.354854 3798194 disk_info.go:125] begin check local disk info of client I1122 05:27:02.358764 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:27:02.358773 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000417900 0xc000417940] E1122 05:27:06.412967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:27:06.412991 3798194 memory.go:184] no items to output this cycle I1122 05:27:06.413154 3798194 cpu.go:275] no items to output this cycle E1122 05:27:16.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:27:16.413266 3798194 memory.go:184] no items to output this cycle I1122 05:27:16.413409 3798194 cpu.go:275] no items to output this cycle I1122 05:27:17.638206 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:27:17.638214 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:27:26.413248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:27:26.413284 3798194 memory.go:191] Add success. I1122 05:27:26.417531 3798194 cpu.go:282] Add success. I1122 05:27:26.441234 3798194 net.go:642] Add success. I1122 05:27:26.442745 3798194 net.go:778] primary dev: ETH0 I1122 05:27:26.442761 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:27:26.442776 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:27:29.450006 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:27:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:27:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:27:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:27:36.411983 3798194 memory.go:184] no items to output this cycle I1122 05:27:36.412139 3798194 cpu.go:275] no items to output this cycle E1122 05:27:46.412338 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:27:46.412360 3798194 memory.go:184] no items to output this cycle I1122 05:27:46.412514 3798194 cpu.go:275] no items to output this cycle E1122 05:27:56.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:27:56.412881 3798194 memory.go:191] Add success. W1122 05:27:56.412915 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:27:56.412927 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:27:56.412930 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:27:56.417138 3798194 cpu.go:282] Add success. I1122 05:27:56.441625 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:27:56.442207 3798194 net.go:642] Add success. I1122 05:27:56.443672 3798194 net.go:778] primary dev: ETH0 I1122 05:27:56.443687 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:27:56.443700 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:27:57.451031 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:27:57.451261 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:27:57.451272 3798194 disk_worker.go:723] disk space is not compliant W1122 05:27:57.451275 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:27:57.460060 3798194 disk_worker.go:509] system disk:vda1 I1122 05:27:57.460101 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:27:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:27:59.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:27:59.450467 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:27:59.450492 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:27:59.471007 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:28:02.358863 3798194 disk_info.go:125] begin check local disk info of client I1122 05:28:02.362741 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:28:02.362750 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d6b80 0xc0003d6bc0] E1122 05:28:06.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:28:06.411986 3798194 memory.go:184] no items to output this cycle I1122 05:28:06.412143 3798194 cpu.go:275] no items to output this cycle E1122 05:28:16.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:28:16.413270 3798194 memory.go:184] no items to output this cycle I1122 05:28:16.413413 3798194 cpu.go:275] no items to output this cycle E1122 05:28:26.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:28:26.412262 3798194 memory.go:191] Add success. I1122 05:28:26.416604 3798194 cpu.go:282] Add success. I1122 05:28:26.441289 3798194 net.go:642] Add success. I1122 05:28:26.442799 3798194 net.go:778] primary dev: ETH0 I1122 05:28:26.442813 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:28:26.442828 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:28:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:28:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:28:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:28:36.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:28:36.411981 3798194 memory.go:184] no items to output this cycle I1122 05:28:36.412140 3798194 cpu.go:275] no items to output this cycle E1122 05:28:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:28:46.411979 3798194 memory.go:184] no items to output this cycle I1122 05:28:46.412132 3798194 cpu.go:275] no items to output this cycle E1122 05:28:56.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:28:56.413269 3798194 memory.go:191] Add success. W1122 05:28:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:28:56.413310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:28:56.413313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:28:56.417483 3798194 cpu.go:282] Add success. I1122 05:28:56.441877 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:28:56.442384 3798194 net.go:642] Add success. I1122 05:28:56.443811 3798194 net.go:778] primary dev: ETH0 I1122 05:28:56.443825 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:28:56.443839 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:28:57.451180 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:28:57.451381 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:28:57.451391 3798194 disk_worker.go:723] disk space is not compliant W1122 05:28:57.451394 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:28:57.459739 3798194 disk_worker.go:509] system disk:vda1 I1122 05:28:57.459783 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:28:58.446373 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:28:59.450386 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:28:59.450462 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:28:59.450487 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:28:59.471019 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:29:02.366913 3798194 disk_info.go:125] begin check local disk info of client I1122 05:29:02.370752 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:29:02.370760 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000468fc0 0xc000469000] E1122 05:29:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:29:06.411975 3798194 memory.go:184] no items to output this cycle I1122 05:29:06.412127 3798194 cpu.go:275] no items to output this cycle E1122 05:29:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:29:16.411978 3798194 memory.go:184] no items to output this cycle I1122 05:29:16.412138 3798194 cpu.go:275] no items to output this cycle E1122 05:29:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:29:26.411953 3798194 memory.go:191] Add success. I1122 05:29:26.412304 3798194 cpu.go:282] Add success. I1122 05:29:26.441902 3798194 net.go:642] Add success. I1122 05:29:26.446188 3798194 net.go:778] primary dev: ETH0 I1122 05:29:26.446204 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:29:26.446221 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:29:29.450006 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:29:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:29:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:29:36.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:29:36.412253 3798194 memory.go:184] no items to output this cycle I1122 05:29:36.412408 3798194 cpu.go:275] no items to output this cycle E1122 05:29:46.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:29:46.412260 3798194 memory.go:184] no items to output this cycle I1122 05:29:46.412410 3798194 cpu.go:275] no items to output this cycle E1122 05:29:56.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:29:56.413274 3798194 memory.go:191] Add success. W1122 05:29:56.413307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:29:56.413318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:29:56.413322 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:29:56.417888 3798194 cpu.go:282] Add success. I1122 05:29:56.441628 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:29:56.442826 3798194 net.go:642] Add success. I1122 05:29:56.444297 3798194 net.go:778] primary dev: ETH0 I1122 05:29:56.444311 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:29:56.444324 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:29:57.450992 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:29:57.451227 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:29:57.451239 3798194 disk_worker.go:723] disk space is not compliant W1122 05:29:57.451242 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:29:57.459985 3798194 disk_worker.go:509] system disk:vda1 I1122 05:29:57.460026 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:29:58.445947 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:29:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:29:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:29:59.450130 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:29:59.471646 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:30:02.370860 3798194 disk_info.go:125] begin check local disk info of client I1122 05:30:02.374699 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:30:02.374707 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004b2a40 0xc0004b2a80] E1122 05:30:06.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:30:06.411993 3798194 memory.go:184] no items to output this cycle I1122 05:30:06.412154 3798194 cpu.go:275] no items to output this cycle E1122 05:30:16.411973 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:30:16.411995 3798194 memory.go:184] no items to output this cycle I1122 05:30:16.412180 3798194 cpu.go:275] no items to output this cycle I1122 05:30:17.638373 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:30:17.638382 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:30:26.413401 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:30:26.413437 3798194 memory.go:191] Add success. I1122 05:30:26.417603 3798194 cpu.go:282] Add success. I1122 05:30:26.441265 3798194 net.go:642] Add success. I1122 05:30:26.442840 3798194 net.go:778] primary dev: ETH0 I1122 05:30:26.442856 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:30:26.442869 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:30:29.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:30:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:30:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:30:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:30:36.413256 3798194 memory.go:184] no items to output this cycle I1122 05:30:36.413402 3798194 cpu.go:275] no items to output this cycle E1122 05:30:46.413252 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:30:46.413275 3798194 memory.go:184] no items to output this cycle I1122 05:30:46.413427 3798194 cpu.go:275] no items to output this cycle E1122 05:30:56.412867 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:30:56.412899 3798194 memory.go:191] Add success. W1122 05:30:56.412933 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:30:56.412944 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:30:56.412948 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:30:56.417205 3798194 cpu.go:282] Add success. I1122 05:30:56.444595 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:30:56.445156 3798194 net.go:642] Add success. I1122 05:30:56.446742 3798194 net.go:778] primary dev: ETH0 I1122 05:30:56.446757 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:30:56.446771 3798194 net.go:659] Add eni metrics success, count: 7 E1122 05:30:57.451078 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 05:30:57.451092 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:30:57.451266 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:30:57.451323 3798194 disk_worker.go:723] disk space is not compliant W1122 05:30:57.451327 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:30:57.460391 3798194 disk_worker.go:509] system disk:vda1 I1122 05:30:57.460429 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 05:30:58.448264 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 05:30:58.448279 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 05:30:59.448790 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 05:30:59.449865 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:30:59.449918 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:30:59.449936 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:30:59.471465 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:31:02.378145 3798194 disk_info.go:125] begin check local disk info of client I1122 05:31:02.381985 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:31:02.381994 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000343940 0xc000343980] E1122 05:31:06.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:31:06.413262 3798194 memory.go:184] no items to output this cycle I1122 05:31:06.413408 3798194 cpu.go:275] no items to output this cycle E1122 05:31:16.412711 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:31:16.412736 3798194 memory.go:184] no items to output this cycle I1122 05:31:16.412881 3798194 cpu.go:275] no items to output this cycle E1122 05:31:26.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:31:26.411963 3798194 memory.go:191] Add success. I1122 05:31:26.412321 3798194 cpu.go:282] Add success. I1122 05:31:26.441927 3798194 net.go:642] Add success. I1122 05:31:26.445838 3798194 net.go:778] primary dev: ETH0 I1122 05:31:26.445853 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:31:26.445867 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:31:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:31:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:31:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:31:36.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:31:36.413264 3798194 memory.go:184] no items to output this cycle I1122 05:31:36.413407 3798194 cpu.go:275] no items to output this cycle E1122 05:31:46.413252 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:31:46.413282 3798194 memory.go:184] no items to output this cycle I1122 05:31:46.413441 3798194 cpu.go:275] no items to output this cycle E1122 05:31:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:31:56.411959 3798194 memory.go:191] Add success. W1122 05:31:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:31:56.412008 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:31:56.412013 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:31:56.412391 3798194 cpu.go:282] Add success. I1122 05:31:56.441912 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:31:56.442492 3798194 net.go:642] Add success. I1122 05:31:56.446280 3798194 net.go:778] primary dev: ETH0 I1122 05:31:56.446294 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:31:56.446306 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:31:57.453819 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:31:57.454004 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:31:57.454061 3798194 disk_worker.go:723] disk space is not compliant W1122 05:31:57.454064 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:31:57.465311 3798194 disk_worker.go:509] system disk:vda1 I1122 05:31:57.465351 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:31:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:31:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:31:59.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:31:59.450149 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:31:59.470671 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:32:02.385983 3798194 disk_info.go:125] begin check local disk info of client I1122 05:32:02.389974 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:32:02.389984 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fe3c0 0xc0003fe400] E1122 05:32:06.412087 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:32:06.412128 3798194 memory.go:184] no items to output this cycle I1122 05:32:06.412293 3798194 cpu.go:275] no items to output this cycle E1122 05:32:16.412426 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:32:16.412457 3798194 memory.go:184] no items to output this cycle I1122 05:32:16.412724 3798194 cpu.go:275] no items to output this cycle E1122 05:32:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:32:26.411955 3798194 memory.go:191] Add success. I1122 05:32:26.412344 3798194 cpu.go:282] Add success. I1122 05:32:26.441946 3798194 net.go:642] Add success. I1122 05:32:26.445873 3798194 net.go:778] primary dev: ETH0 I1122 05:32:26.445889 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:32:26.445903 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:32:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:32:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:32:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:32:36.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:32:36.413251 3798194 memory.go:184] no items to output this cycle I1122 05:32:36.413396 3798194 cpu.go:275] no items to output this cycle E1122 05:32:46.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:32:46.413253 3798194 memory.go:184] no items to output this cycle I1122 05:32:46.413399 3798194 cpu.go:275] no items to output this cycle E1122 05:32:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:32:56.411953 3798194 memory.go:191] Add success. W1122 05:32:56.411981 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:32:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:32:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:32:56.412341 3798194 cpu.go:282] Add success. I1122 05:32:56.441930 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:32:56.442531 3798194 net.go:642] Add success. I1122 05:32:56.446439 3798194 net.go:778] primary dev: ETH0 I1122 05:32:56.446454 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:32:56.446465 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:32:57.450733 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:32:57.450892 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:32:57.450942 3798194 disk_worker.go:723] disk space is not compliant W1122 05:32:57.450945 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:32:57.459253 3798194 disk_worker.go:509] system disk:vda1 I1122 05:32:57.459291 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:32:58.446023 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:32:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:32:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:32:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:32:59.471645 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:33:02.394674 3798194 disk_info.go:125] begin check local disk info of client I1122 05:33:02.398585 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:33:02.398595 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000367440 0xc000367480] E1122 05:33:06.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:33:06.411989 3798194 memory.go:184] no items to output this cycle I1122 05:33:06.412147 3798194 cpu.go:275] no items to output this cycle E1122 05:33:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:33:16.411974 3798194 memory.go:184] no items to output this cycle I1122 05:33:16.412133 3798194 cpu.go:275] no items to output this cycle I1122 05:33:17.638542 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:33:17.638551 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:33:26.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:33:26.412884 3798194 memory.go:191] Add success. I1122 05:33:26.416922 3798194 cpu.go:282] Add success. I1122 05:33:26.441634 3798194 net.go:642] Add success. I1122 05:33:26.443144 3798194 net.go:778] primary dev: ETH0 I1122 05:33:26.443160 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:33:26.443174 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:33:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:33:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:33:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:33:36.412070 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:33:36.412095 3798194 memory.go:184] no items to output this cycle I1122 05:33:36.412443 3798194 cpu.go:275] no items to output this cycle E1122 05:33:46.413063 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:33:46.413085 3798194 memory.go:184] no items to output this cycle I1122 05:33:46.413258 3798194 cpu.go:275] no items to output this cycle E1122 05:33:56.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:33:56.413275 3798194 memory.go:191] Add success. W1122 05:33:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:33:56.413319 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:33:56.413323 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:33:56.417835 3798194 cpu.go:282] Add success. I1122 05:33:56.441231 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:33:56.441728 3798194 net.go:642] Add success. I1122 05:33:56.443287 3798194 net.go:778] primary dev: ETH0 I1122 05:33:56.443301 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:33:56.443314 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:33:57.451324 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:33:57.451536 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:33:57.451547 3798194 disk_worker.go:723] disk space is not compliant W1122 05:33:57.451550 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:33:57.459899 3798194 disk_worker.go:509] system disk:vda1 I1122 05:33:57.459941 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:33:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:33:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:33:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:33:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:33:59.471663 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:34:02.403348 3798194 disk_info.go:125] begin check local disk info of client I1122 05:34:02.407278 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:34:02.407287 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000417c40 0xc000417c80] E1122 05:34:06.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:34:06.412879 3798194 memory.go:184] no items to output this cycle I1122 05:34:06.413019 3798194 cpu.go:275] no items to output this cycle E1122 05:34:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:34:16.411980 3798194 memory.go:184] no items to output this cycle I1122 05:34:16.412147 3798194 cpu.go:275] no items to output this cycle E1122 05:34:26.412183 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:34:26.412218 3798194 memory.go:191] Add success. I1122 05:34:26.416659 3798194 cpu.go:282] Add success. I1122 05:34:26.442229 3798194 net.go:642] Add success. I1122 05:34:26.443900 3798194 net.go:778] primary dev: ETH0 I1122 05:34:26.443914 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:34:26.443928 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:34:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:34:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:34:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:34:36.412222 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:34:36.412245 3798194 memory.go:184] no items to output this cycle I1122 05:34:36.412400 3798194 cpu.go:275] no items to output this cycle E1122 05:34:46.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:34:46.411994 3798194 memory.go:184] no items to output this cycle I1122 05:34:46.412157 3798194 cpu.go:275] no items to output this cycle E1122 05:34:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:34:56.411952 3798194 memory.go:191] Add success. W1122 05:34:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:34:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:34:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:34:56.412535 3798194 cpu.go:282] Add success. I1122 05:34:56.440956 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:34:56.441544 3798194 net.go:642] Add success. I1122 05:34:56.445743 3798194 net.go:778] primary dev: ETH0 I1122 05:34:56.445757 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:34:56.445771 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:34:57.450942 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:34:57.451184 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:34:57.451197 3798194 disk_worker.go:723] disk space is not compliant W1122 05:34:57.451200 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:34:57.459971 3798194 disk_worker.go:509] system disk:vda1 I1122 05:34:57.460013 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:34:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:34:59.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:34:59.450124 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:34:59.450151 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:34:59.471688 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:35:02.407391 3798194 disk_info.go:125] begin check local disk info of client I1122 05:35:02.411486 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:35:02.411498 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e1480 0xc0002e14c0] E1122 05:35:06.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:35:06.412878 3798194 memory.go:184] no items to output this cycle I1122 05:35:06.413027 3798194 cpu.go:275] no items to output this cycle E1122 05:35:16.412945 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:35:16.412969 3798194 memory.go:184] no items to output this cycle I1122 05:35:16.413137 3798194 cpu.go:275] no items to output this cycle E1122 05:35:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:35:26.413263 3798194 memory.go:191] Add success. I1122 05:35:26.417392 3798194 cpu.go:282] Add success. I1122 05:35:26.442171 3798194 net.go:642] Add success. I1122 05:35:26.443784 3798194 net.go:778] primary dev: ETH0 I1122 05:35:26.443800 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:35:26.443812 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:35:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:35:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:35:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:35:36.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:35:36.413252 3798194 memory.go:184] no items to output this cycle I1122 05:35:36.413396 3798194 cpu.go:275] no items to output this cycle E1122 05:35:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:35:46.411977 3798194 memory.go:184] no items to output this cycle I1122 05:35:46.412128 3798194 cpu.go:275] no items to output this cycle E1122 05:35:56.411907 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:35:56.411942 3798194 memory.go:191] Add success. W1122 05:35:56.411975 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:35:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:35:56.411990 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:35:56.412342 3798194 cpu.go:282] Add success. I1122 05:35:56.441548 3798194 event_worker.go:152] Polling the log file for events... I1122 05:35:56.444856 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:35:56.445399 3798194 net.go:642] Add success. I1122 05:35:56.449507 3798194 net.go:778] primary dev: ETH0 I1122 05:35:56.449521 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:35:56.449536 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:35:56.449553 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795295156, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 05:35:56.461437 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 05:35:57.449292 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 05:35:57.449305 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:35:57.449486 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:35:57.449543 3798194 disk_worker.go:723] disk space is not compliant W1122 05:35:57.449546 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:35:57.452219 3798194 disk_worker.go:509] system disk:vda1 I1122 05:35:57.452257 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 05:35:58.448550 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 05:35:58.448566 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 05:35:59.448660 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 05:35:59.449735 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:35:59.449789 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:35:59.449815 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:35:59.471352 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:36:02.413322 3798194 disk_info.go:125] begin check local disk info of client I1122 05:36:02.417307 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:36:02.417316 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000457980 0xc0004579c0] E1122 05:36:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:36:06.412257 3798194 memory.go:184] no items to output this cycle I1122 05:36:06.412399 3798194 cpu.go:275] no items to output this cycle E1122 05:36:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:36:16.411979 3798194 memory.go:184] no items to output this cycle I1122 05:36:16.412138 3798194 cpu.go:275] no items to output this cycle I1122 05:36:17.638710 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:36:17.638719 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:36:26.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:36:26.412897 3798194 memory.go:191] Add success. I1122 05:36:26.417108 3798194 cpu.go:282] Add success. I1122 05:36:26.441659 3798194 net.go:642] Add success. I1122 05:36:26.443099 3798194 net.go:778] primary dev: ETH0 I1122 05:36:26.443129 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:36:26.443144 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:36:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:36:29.450492 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:36:29.450519 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:36:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:36:36.412265 3798194 memory.go:184] no items to output this cycle I1122 05:36:36.412407 3798194 cpu.go:275] no items to output this cycle E1122 05:36:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:36:46.411979 3798194 memory.go:184] no items to output this cycle I1122 05:36:46.412137 3798194 cpu.go:275] no items to output this cycle E1122 05:36:56.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:36:56.412889 3798194 memory.go:191] Add success. W1122 05:36:56.412918 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:36:56.412929 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:36:56.412933 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:36:56.417035 3798194 cpu.go:282] Add success. I1122 05:36:56.441598 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:36:56.442085 3798194 net.go:642] Add success. I1122 05:36:56.443654 3798194 net.go:778] primary dev: ETH0 I1122 05:36:56.443669 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:36:56.443681 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:36:57.451086 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:36:57.451272 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:36:57.451327 3798194 disk_worker.go:723] disk space is not compliant W1122 05:36:57.451331 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:36:57.459905 3798194 disk_worker.go:509] system disk:vda1 I1122 05:36:57.459951 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:36:58.445691 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:36:59.449671 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:36:59.449755 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:36:59.449782 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:36:59.471349 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:37:02.417858 3798194 disk_info.go:125] begin check local disk info of client I1122 05:37:02.421801 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:37:02.421811 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ecd80 0xc0003ecdc0] E1122 05:37:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:37:06.411980 3798194 memory.go:184] no items to output this cycle I1122 05:37:06.412133 3798194 cpu.go:275] no items to output this cycle E1122 05:37:16.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:37:16.412272 3798194 memory.go:184] no items to output this cycle I1122 05:37:16.412423 3798194 cpu.go:275] no items to output this cycle E1122 05:37:26.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:37:26.413259 3798194 memory.go:191] Add success. I1122 05:37:26.417541 3798194 cpu.go:282] Add success. I1122 05:37:26.442105 3798194 net.go:642] Add success. I1122 05:37:26.443582 3798194 net.go:778] primary dev: ETH0 I1122 05:37:26.443597 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:37:26.443610 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:37:29.450192 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:37:29.450277 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:37:29.450309 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:37:36.412150 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:37:36.412173 3798194 memory.go:184] no items to output this cycle I1122 05:37:36.412321 3798194 cpu.go:275] no items to output this cycle E1122 05:37:46.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:37:46.411984 3798194 memory.go:184] no items to output this cycle I1122 05:37:46.412144 3798194 cpu.go:275] no items to output this cycle E1122 05:37:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:37:56.412268 3798194 memory.go:191] Add success. W1122 05:37:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:37:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:37:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:37:56.416298 3798194 cpu.go:282] Add success. I1122 05:37:56.441708 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:37:56.442211 3798194 net.go:642] Add success. I1122 05:37:56.443623 3798194 net.go:778] primary dev: ETH0 I1122 05:37:56.443638 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:37:56.443652 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:37:57.451986 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:37:57.452229 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:37:57.452241 3798194 disk_worker.go:723] disk space is not compliant W1122 05:37:57.452244 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:37:57.460991 3798194 disk_worker.go:509] system disk:vda1 I1122 05:37:57.461034 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:37:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:37:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:37:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:37:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:37:59.471684 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:38:02.422881 3798194 disk_info.go:125] begin check local disk info of client I1122 05:38:02.426775 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:38:02.426784 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000326340 0xc000326380] E1122 05:38:06.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:38:06.412257 3798194 memory.go:184] no items to output this cycle I1122 05:38:06.412401 3798194 cpu.go:275] no items to output this cycle E1122 05:38:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:38:16.411972 3798194 memory.go:184] no items to output this cycle I1122 05:38:16.412129 3798194 cpu.go:275] no items to output this cycle E1122 05:38:26.412101 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:38:26.412158 3798194 memory.go:191] Add success. I1122 05:38:26.416843 3798194 cpu.go:282] Add success. I1122 05:38:26.441550 3798194 net.go:642] Add success. I1122 05:38:26.443104 3798194 net.go:778] primary dev: ETH0 I1122 05:38:26.443129 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:38:26.443141 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:38:29.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:38:29.450462 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:38:29.450488 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:38:36.412935 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:38:36.412957 3798194 memory.go:184] no items to output this cycle I1122 05:38:36.413097 3798194 cpu.go:275] no items to output this cycle E1122 05:38:46.413350 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:38:46.413437 3798194 memory.go:184] no items to output this cycle I1122 05:38:46.413788 3798194 cpu.go:275] no items to output this cycle E1122 05:38:56.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:38:56.412890 3798194 memory.go:191] Add success. W1122 05:38:56.412932 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:38:56.412945 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:38:56.412949 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:38:56.417299 3798194 cpu.go:282] Add success. I1122 05:38:56.441666 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:38:56.442173 3798194 net.go:642] Add success. I1122 05:38:56.443674 3798194 net.go:778] primary dev: ETH0 I1122 05:38:56.443689 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:38:56.443702 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:38:57.450607 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:38:57.450808 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:38:57.450818 3798194 disk_worker.go:723] disk space is not compliant W1122 05:38:57.450821 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:38:57.459212 3798194 disk_worker.go:509] system disk:vda1 I1122 05:38:57.459251 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:38:58.446373 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:38:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:38:59.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:38:59.450502 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:38:59.471051 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:39:02.427286 3798194 disk_info.go:125] begin check local disk info of client I1122 05:39:02.431179 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:39:02.431188 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004044c0 0xc000404500] E1122 05:39:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:39:06.412254 3798194 memory.go:184] no items to output this cycle I1122 05:39:06.412407 3798194 cpu.go:275] no items to output this cycle E1122 05:39:16.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:39:16.412876 3798194 memory.go:184] no items to output this cycle I1122 05:39:16.413020 3798194 cpu.go:275] no items to output this cycle I1122 05:39:17.638881 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:39:17.638890 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:39:26.412807 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:39:26.412840 3798194 memory.go:191] Add success. I1122 05:39:26.416898 3798194 cpu.go:282] Add success. I1122 05:39:26.441640 3798194 net.go:642] Add success. I1122 05:39:26.447841 3798194 net.go:778] primary dev: ETH0 I1122 05:39:26.447862 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:39:26.447875 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:39:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:39:29.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:39:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:39:36.412843 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:39:36.412864 3798194 memory.go:184] no items to output this cycle I1122 05:39:36.413006 3798194 cpu.go:275] no items to output this cycle E1122 05:39:46.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:39:46.412878 3798194 memory.go:184] no items to output this cycle I1122 05:39:46.413034 3798194 cpu.go:275] no items to output this cycle E1122 05:39:56.412645 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:39:56.412680 3798194 memory.go:191] Add success. W1122 05:39:56.412707 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:39:56.412719 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:39:56.412722 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:39:56.417052 3798194 cpu.go:282] Add success. I1122 05:39:56.441416 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:39:56.441950 3798194 net.go:642] Add success. I1122 05:39:56.443318 3798194 net.go:778] primary dev: ETH0 I1122 05:39:56.443332 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:39:56.443345 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:39:57.450946 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:39:57.451167 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:39:57.451180 3798194 disk_worker.go:723] disk space is not compliant W1122 05:39:57.451183 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:39:57.460365 3798194 disk_worker.go:509] system disk:vda1 I1122 05:39:57.460405 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:39:58.446193 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:39:59.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:39:59.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:39:59.450497 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:39:59.471012 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:40:02.431254 3798194 disk_info.go:125] begin check local disk info of client I1122 05:40:02.435168 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:40:02.435178 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004c3440 0xc0004c3480] E1122 05:40:06.412647 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:40:06.412670 3798194 memory.go:184] no items to output this cycle I1122 05:40:06.412824 3798194 cpu.go:275] no items to output this cycle E1122 05:40:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:40:16.412256 3798194 memory.go:184] no items to output this cycle I1122 05:40:16.412403 3798194 cpu.go:275] no items to output this cycle E1122 05:40:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:40:26.411952 3798194 memory.go:191] Add success. I1122 05:40:26.412315 3798194 cpu.go:282] Add success. I1122 05:40:26.441938 3798194 net.go:642] Add success. I1122 05:40:26.446123 3798194 net.go:778] primary dev: ETH0 I1122 05:40:26.446138 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:40:26.446151 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:40:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:40:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:40:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:40:36.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:40:36.413262 3798194 memory.go:184] no items to output this cycle I1122 05:40:36.413409 3798194 cpu.go:275] no items to output this cycle E1122 05:40:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:40:46.411978 3798194 memory.go:184] no items to output this cycle I1122 05:40:46.412140 3798194 cpu.go:275] no items to output this cycle E1122 05:40:56.411908 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:40:56.411942 3798194 memory.go:191] Add success. W1122 05:40:56.411969 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:40:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:40:56.411984 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:40:56.412337 3798194 cpu.go:282] Add success. I1122 05:40:56.444688 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:40:56.445192 3798194 net.go:642] Add success. I1122 05:40:56.449468 3798194 net.go:778] primary dev: ETH0 I1122 05:40:56.449482 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:40:56.449493 3798194 net.go:659] Add eni metrics success, count: 7 E1122 05:40:57.450850 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 05:40:57.450865 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:40:57.451075 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:40:57.451084 3798194 disk_worker.go:723] disk space is not compliant W1122 05:40:57.451087 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:40:57.460028 3798194 disk_worker.go:509] system disk:vda1 I1122 05:40:57.460072 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 05:40:58.448084 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 05:40:58.448099 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 05:40:59.448624 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 05:40:59.449701 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:40:59.449754 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:40:59.449777 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:40:59.471301 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:41:02.435263 3798194 disk_info.go:125] begin check local disk info of client I1122 05:41:02.439191 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:41:02.439201 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b84c0 0xc0003b8500] E1122 05:41:06.412184 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:41:06.412206 3798194 memory.go:184] no items to output this cycle I1122 05:41:06.412347 3798194 cpu.go:275] no items to output this cycle E1122 05:41:16.412802 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:41:16.412826 3798194 memory.go:184] no items to output this cycle I1122 05:41:16.412972 3798194 cpu.go:275] no items to output this cycle E1122 05:41:26.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:41:26.412271 3798194 memory.go:191] Add success. I1122 05:41:26.416477 3798194 cpu.go:282] Add success. I1122 05:41:26.441308 3798194 net.go:642] Add success. I1122 05:41:26.442953 3798194 net.go:778] primary dev: ETH0 I1122 05:41:26.442968 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:41:26.442981 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:41:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:41:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:41:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:41:36.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:41:36.412885 3798194 memory.go:184] no items to output this cycle I1122 05:41:36.413032 3798194 cpu.go:275] no items to output this cycle E1122 05:41:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:41:46.411977 3798194 memory.go:184] no items to output this cycle I1122 05:41:46.412133 3798194 cpu.go:275] no items to output this cycle E1122 05:41:56.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:41:56.411965 3798194 memory.go:191] Add success. W1122 05:41:56.411992 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:41:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:41:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:41:56.412351 3798194 cpu.go:282] Add success. I1122 05:41:56.441819 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:41:56.442415 3798194 net.go:642] Add success. I1122 05:41:56.446377 3798194 net.go:778] primary dev: ETH0 I1122 05:41:56.446391 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:41:56.446405 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:41:57.450780 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:41:57.450936 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:41:57.450987 3798194 disk_worker.go:723] disk space is not compliant W1122 05:41:57.450990 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:41:57.459770 3798194 disk_worker.go:509] system disk:vda1 I1122 05:41:57.459813 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:41:58.446458 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:41:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:41:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:41:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:41:59.471673 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:42:02.439987 3798194 disk_info.go:125] begin check local disk info of client I1122 05:42:02.443889 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:42:02.443898 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b86c0 0xc0003b8700] E1122 05:42:06.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:42:06.412249 3798194 memory.go:184] no items to output this cycle I1122 05:42:06.412397 3798194 cpu.go:275] no items to output this cycle E1122 05:42:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:42:16.411983 3798194 memory.go:184] no items to output this cycle I1122 05:42:16.412142 3798194 cpu.go:275] no items to output this cycle I1122 05:42:17.639065 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:42:17.639074 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:42:26.412868 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:42:26.412903 3798194 memory.go:191] Add success. I1122 05:42:26.417248 3798194 cpu.go:282] Add success. I1122 05:42:26.441750 3798194 net.go:642] Add success. I1122 05:42:26.443160 3798194 net.go:778] primary dev: ETH0 I1122 05:42:26.443182 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:42:26.443196 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:42:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:42:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:42:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:42:36.413171 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:42:36.413192 3798194 memory.go:184] no items to output this cycle I1122 05:42:36.413333 3798194 cpu.go:275] no items to output this cycle E1122 05:42:46.412224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:42:46.412246 3798194 memory.go:184] no items to output this cycle I1122 05:42:46.412393 3798194 cpu.go:275] no items to output this cycle E1122 05:42:56.412844 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:42:56.412878 3798194 memory.go:191] Add success. W1122 05:42:56.412905 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:42:56.412917 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:42:56.412920 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:42:56.417486 3798194 cpu.go:282] Add success. I1122 05:42:56.441602 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:42:56.442092 3798194 net.go:642] Add success. I1122 05:42:56.443535 3798194 net.go:778] primary dev: ETH0 I1122 05:42:56.443549 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:42:56.443562 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:42:57.446472 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:42:57.446669 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:42:57.446731 3798194 disk_worker.go:723] disk space is not compliant W1122 05:42:57.446734 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:42:57.449505 3798194 disk_worker.go:509] system disk:vda1 I1122 05:42:57.449545 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:42:58.446367 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:42:59.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:42:59.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:42:59.450489 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:42:59.471021 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:43:02.444378 3798194 disk_info.go:125] begin check local disk info of client I1122 05:43:02.448244 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:43:02.448252 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030f6c0 0xc00030f700] E1122 05:43:06.411230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:43:06.411252 3798194 memory.go:184] no items to output this cycle I1122 05:43:06.411401 3798194 cpu.go:275] no items to output this cycle E1122 05:43:16.413677 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:43:16.413700 3798194 memory.go:184] no items to output this cycle I1122 05:43:16.413849 3798194 cpu.go:275] no items to output this cycle E1122 05:43:26.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:43:26.412893 3798194 memory.go:191] Add success. I1122 05:43:26.417104 3798194 cpu.go:282] Add success. I1122 05:43:26.441613 3798194 net.go:642] Add success. I1122 05:43:26.443073 3798194 net.go:778] primary dev: ETH0 I1122 05:43:26.443089 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:43:26.443102 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:43:29.449855 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:43:29.449927 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:43:29.449952 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:43:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:43:36.411976 3798194 memory.go:184] no items to output this cycle I1122 05:43:36.412124 3798194 cpu.go:275] no items to output this cycle E1122 05:43:46.413293 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:43:46.413321 3798194 memory.go:184] no items to output this cycle I1122 05:43:46.413526 3798194 cpu.go:275] no items to output this cycle E1122 05:43:56.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:43:56.412263 3798194 memory.go:191] Add success. W1122 05:43:56.412289 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:43:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:43:56.412303 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:43:56.416385 3798194 cpu.go:282] Add success. I1122 05:43:56.441775 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:43:56.442351 3798194 net.go:642] Add success. I1122 05:43:56.443721 3798194 net.go:778] primary dev: ETH0 I1122 05:43:56.443735 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:43:56.443749 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:43:57.451233 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:43:57.451434 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:43:57.451445 3798194 disk_worker.go:723] disk space is not compliant W1122 05:43:57.451448 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:43:57.459795 3798194 disk_worker.go:509] system disk:vda1 I1122 05:43:57.459837 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:43:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:43:59.450188 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:43:59.450267 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:43:59.450291 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:43:59.470724 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:44:02.449023 3798194 disk_info.go:125] begin check local disk info of client I1122 05:44:02.452940 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:44:02.452949 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000417bc0 0xc000417c00] E1122 05:44:06.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:44:06.412250 3798194 memory.go:184] no items to output this cycle I1122 05:44:06.412592 3798194 cpu.go:275] no items to output this cycle E1122 05:44:16.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:44:16.413264 3798194 memory.go:184] no items to output this cycle I1122 05:44:16.413413 3798194 cpu.go:275] no items to output this cycle E1122 05:44:26.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:44:26.412897 3798194 memory.go:191] Add success. I1122 05:44:26.417182 3798194 cpu.go:282] Add success. I1122 05:44:26.441686 3798194 net.go:642] Add success. I1122 05:44:26.443042 3798194 net.go:778] primary dev: ETH0 I1122 05:44:26.443055 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:44:26.443069 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:44:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:44:29.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:44:29.450149 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:44:36.412751 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:44:36.412773 3798194 memory.go:184] no items to output this cycle I1122 05:44:36.412917 3798194 cpu.go:275] no items to output this cycle E1122 05:44:46.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:44:46.412253 3798194 memory.go:184] no items to output this cycle I1122 05:44:46.412401 3798194 cpu.go:275] no items to output this cycle E1122 05:44:56.412764 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:44:56.412799 3798194 memory.go:191] Add success. W1122 05:44:56.412829 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:44:56.412842 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:44:56.412846 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:44:56.417275 3798194 cpu.go:282] Add success. I1122 05:44:56.441668 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:44:56.442163 3798194 net.go:642] Add success. I1122 05:44:56.443730 3798194 net.go:778] primary dev: ETH0 I1122 05:44:56.443750 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:44:56.443768 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:44:57.451492 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:44:57.451703 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:44:57.451714 3798194 disk_worker.go:723] disk space is not compliant W1122 05:44:57.451717 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:44:57.460358 3798194 disk_worker.go:509] system disk:vda1 I1122 05:44:57.460411 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:44:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:44:59.450105 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:44:59.450195 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:44:59.450224 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:44:59.470693 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:45:02.453410 3798194 disk_info.go:125] begin check local disk info of client I1122 05:45:02.457301 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:45:02.457311 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037efc0 0xc00037f000] E1122 05:45:06.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:45:06.413254 3798194 memory.go:184] no items to output this cycle I1122 05:45:06.413397 3798194 cpu.go:275] no items to output this cycle E1122 05:45:16.412759 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:45:16.412782 3798194 memory.go:184] no items to output this cycle I1122 05:45:16.412931 3798194 cpu.go:275] no items to output this cycle I1122 05:45:17.639224 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:45:17.639233 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:45:26.413075 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:45:26.413128 3798194 memory.go:191] Add success. I1122 05:45:26.417480 3798194 cpu.go:282] Add success. I1122 05:45:26.442013 3798194 net.go:642] Add success. I1122 05:45:26.443341 3798194 net.go:778] primary dev: ETH0 I1122 05:45:26.443357 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:45:26.443370 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:45:29.450008 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:45:29.450084 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:45:29.450122 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:45:36.412869 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:45:36.412890 3798194 memory.go:184] no items to output this cycle I1122 05:45:36.413194 3798194 cpu.go:275] no items to output this cycle E1122 05:45:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:45:46.411971 3798194 memory.go:184] no items to output this cycle I1122 05:45:46.412126 3798194 cpu.go:275] no items to output this cycle E1122 05:45:56.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:45:56.412895 3798194 memory.go:191] Add success. W1122 05:45:56.412925 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:45:56.412936 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:45:56.412939 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:45:56.417069 3798194 cpu.go:282] Add success. I1122 05:45:56.444128 3798194 event_worker.go:152] Polling the log file for events... I1122 05:45:56.444335 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:45:56.444913 3798194 net.go:642] Add success. I1122 05:45:56.446595 3798194 net.go:778] primary dev: ETH0 I1122 05:45:56.446609 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:45:56.446623 3798194 net.go:659] Add eni metrics success, count: 7 E1122 05:45:57.451051 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 05:45:57.451065 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:45:57.451279 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:45:57.451289 3798194 disk_worker.go:723] disk space is not compliant W1122 05:45:57.451292 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:45:57.459839 3798194 disk_worker.go:509] system disk:vda1 I1122 05:45:57.459882 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 05:45:58.448472 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 05:45:58.448487 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 05:45:59.448984 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 05:45:59.450058 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:45:59.450115 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:45:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:45:59.471647 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:46:02.457400 3798194 disk_info.go:125] begin check local disk info of client I1122 05:46:02.461321 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:46:02.461330 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d8fc0 0xc0003d9000] E1122 05:46:06.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:46:06.413249 3798194 memory.go:184] no items to output this cycle I1122 05:46:06.413404 3798194 cpu.go:275] no items to output this cycle E1122 05:46:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:46:16.411974 3798194 memory.go:184] no items to output this cycle I1122 05:46:16.412131 3798194 cpu.go:275] no items to output this cycle E1122 05:46:26.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:46:26.412898 3798194 memory.go:191] Add success. I1122 05:46:26.417171 3798194 cpu.go:282] Add success. I1122 05:46:26.441741 3798194 net.go:642] Add success. I1122 05:46:26.443256 3798194 net.go:778] primary dev: ETH0 I1122 05:46:26.443269 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:46:26.443284 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:46:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:46:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:46:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:46:36.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:46:36.413265 3798194 memory.go:184] no items to output this cycle I1122 05:46:36.413405 3798194 cpu.go:275] no items to output this cycle E1122 05:46:46.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:46:46.411988 3798194 memory.go:184] no items to output this cycle I1122 05:46:46.412147 3798194 cpu.go:275] no items to output this cycle E1122 05:46:56.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:46:56.412269 3798194 memory.go:191] Add success. W1122 05:46:56.412301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:46:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:46:56.412321 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:46:56.416711 3798194 cpu.go:282] Add success. I1122 05:46:56.441126 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:46:56.441676 3798194 net.go:642] Add success. I1122 05:46:56.443216 3798194 net.go:778] primary dev: ETH0 I1122 05:46:56.443230 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:46:56.443243 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:46:57.451448 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:46:57.451783 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:46:57.451841 3798194 disk_worker.go:723] disk space is not compliant W1122 05:46:57.451844 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:46:57.460861 3798194 disk_worker.go:509] system disk:vda1 I1122 05:46:57.460902 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:46:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:46:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:46:59.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:46:59.450500 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:46:59.471017 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:47:02.462098 3798194 disk_info.go:125] begin check local disk info of client I1122 05:47:02.466003 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:47:02.466011 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004879c0 0xc000487a00] E1122 05:47:06.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:47:06.412886 3798194 memory.go:184] no items to output this cycle I1122 05:47:06.413028 3798194 cpu.go:275] no items to output this cycle E1122 05:47:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:47:16.411985 3798194 memory.go:184] no items to output this cycle I1122 05:47:16.412144 3798194 cpu.go:275] no items to output this cycle E1122 05:47:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:47:26.411956 3798194 memory.go:191] Add success. I1122 05:47:26.412319 3798194 cpu.go:282] Add success. I1122 05:47:26.441932 3798194 net.go:642] Add success. I1122 05:47:26.445730 3798194 net.go:778] primary dev: ETH0 I1122 05:47:26.445747 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:47:26.445761 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:47:29.449672 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:47:29.449761 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:47:29.449789 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:47:36.413056 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:47:36.413082 3798194 memory.go:184] no items to output this cycle I1122 05:47:36.413242 3798194 cpu.go:275] no items to output this cycle E1122 05:47:46.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:47:46.413249 3798194 memory.go:184] no items to output this cycle I1122 05:47:46.413406 3798194 cpu.go:275] no items to output this cycle E1122 05:47:56.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:47:56.413268 3798194 memory.go:191] Add success. W1122 05:47:56.413301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:47:56.413313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:47:56.413317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:47:56.417664 3798194 cpu.go:282] Add success. I1122 05:47:56.440971 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:47:56.441560 3798194 net.go:642] Add success. I1122 05:47:56.443339 3798194 net.go:778] primary dev: ETH0 I1122 05:47:56.443353 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:47:56.443365 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:47:57.451309 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:47:57.451518 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:47:57.451528 3798194 disk_worker.go:723] disk space is not compliant W1122 05:47:57.451531 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:47:57.459949 3798194 disk_worker.go:509] system disk:vda1 I1122 05:47:57.459994 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:47:58.446635 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:47:59.449674 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:47:59.449760 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:47:59.449785 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:47:59.471323 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:48:02.466482 3798194 disk_info.go:125] begin check local disk info of client I1122 05:48:02.470428 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:48:02.470436 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003dac00 0xc0003dac40] E1122 05:48:06.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:48:06.412269 3798194 memory.go:184] no items to output this cycle I1122 05:48:06.412413 3798194 cpu.go:275] no items to output this cycle E1122 05:48:16.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:48:16.413263 3798194 memory.go:184] no items to output this cycle I1122 05:48:16.413408 3798194 cpu.go:275] no items to output this cycle I1122 05:48:17.640215 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:48:17.640223 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:48:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:48:26.413380 3798194 memory.go:191] Add success. I1122 05:48:26.420167 3798194 cpu.go:282] Add success. I1122 05:48:26.441257 3798194 net.go:642] Add success. I1122 05:48:26.442597 3798194 net.go:778] primary dev: ETH0 I1122 05:48:26.442611 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:48:26.442625 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:48:29.449674 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:48:29.449752 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:48:29.449781 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:48:36.413040 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:48:36.413062 3798194 memory.go:184] no items to output this cycle I1122 05:48:36.413218 3798194 cpu.go:275] no items to output this cycle E1122 05:48:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:48:46.413255 3798194 memory.go:184] no items to output this cycle I1122 05:48:46.413396 3798194 cpu.go:275] no items to output this cycle E1122 05:48:56.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:48:56.412277 3798194 memory.go:191] Add success. W1122 05:48:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:48:56.412324 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:48:56.412328 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:48:56.416274 3798194 cpu.go:282] Add success. I1122 05:48:56.441707 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:48:56.442328 3798194 net.go:642] Add success. I1122 05:48:56.444492 3798194 net.go:778] primary dev: ETH0 I1122 05:48:56.444508 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:48:56.444522 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:48:57.451186 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:48:57.451366 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:48:57.451422 3798194 disk_worker.go:723] disk space is not compliant W1122 05:48:57.451426 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:48:57.459818 3798194 disk_worker.go:509] system disk:vda1 I1122 05:48:57.459862 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:48:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:48:59.450485 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:48:59.450567 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:48:59.450591 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:48:59.471095 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:49:02.471511 3798194 disk_info.go:125] begin check local disk info of client I1122 05:49:02.475453 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:49:02.475462 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002fe880 0xc0002fe8c0] E1122 05:49:06.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:49:06.412259 3798194 memory.go:184] no items to output this cycle I1122 05:49:06.412400 3798194 cpu.go:275] no items to output this cycle E1122 05:49:16.412248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:49:16.412271 3798194 memory.go:184] no items to output this cycle I1122 05:49:16.412414 3798194 cpu.go:275] no items to output this cycle E1122 05:49:26.412030 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:49:26.412066 3798194 memory.go:191] Add success. I1122 05:49:26.416394 3798194 cpu.go:282] Add success. I1122 05:49:26.441935 3798194 net.go:642] Add success. I1122 05:49:26.443416 3798194 net.go:778] primary dev: ETH0 I1122 05:49:26.443431 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:49:26.443447 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:49:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:49:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:49:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:49:36.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:49:36.413254 3798194 memory.go:184] no items to output this cycle I1122 05:49:36.413398 3798194 cpu.go:275] no items to output this cycle E1122 05:49:46.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:49:46.413247 3798194 memory.go:184] no items to output this cycle I1122 05:49:46.413399 3798194 cpu.go:275] no items to output this cycle E1122 05:49:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:49:56.411956 3798194 memory.go:191] Add success. W1122 05:49:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:49:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:49:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:49:56.412513 3798194 cpu.go:282] Add success. I1122 05:49:56.440978 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:49:56.441561 3798194 net.go:642] Add success. I1122 05:49:56.445454 3798194 net.go:778] primary dev: ETH0 I1122 05:49:56.445468 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:49:56.445480 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:49:57.450955 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:49:57.451185 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:49:57.451196 3798194 disk_worker.go:723] disk space is not compliant W1122 05:49:57.451199 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:49:57.462557 3798194 disk_worker.go:509] system disk:vda1 I1122 05:49:57.462598 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:49:58.446606 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:49:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:49:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:49:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:49:59.471663 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:50:02.475561 3798194 disk_info.go:125] begin check local disk info of client I1122 05:50:02.479492 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:50:02.479501 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e0c0 0xc00030e100] E1122 05:50:06.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:50:06.412265 3798194 memory.go:184] no items to output this cycle I1122 05:50:06.412407 3798194 cpu.go:275] no items to output this cycle E1122 05:50:16.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:50:16.412885 3798194 memory.go:184] no items to output this cycle I1122 05:50:16.413033 3798194 cpu.go:275] no items to output this cycle E1122 05:50:26.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:50:26.412266 3798194 memory.go:191] Add success. I1122 05:50:26.416319 3798194 cpu.go:282] Add success. I1122 05:50:26.441727 3798194 net.go:642] Add success. I1122 05:50:26.443163 3798194 net.go:778] primary dev: ETH0 I1122 05:50:26.443178 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:50:26.443189 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:50:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:50:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:50:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:50:36.412844 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:50:36.412865 3798194 memory.go:184] no items to output this cycle I1122 05:50:36.413009 3798194 cpu.go:275] no items to output this cycle E1122 05:50:46.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:50:46.412249 3798194 memory.go:184] no items to output this cycle I1122 05:50:46.412393 3798194 cpu.go:275] no items to output this cycle E1122 05:50:56.411913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:50:56.411946 3798194 memory.go:191] Add success. W1122 05:50:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:50:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:50:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:50:56.412355 3798194 cpu.go:282] Add success. I1122 05:50:56.445015 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:50:56.445581 3798194 net.go:642] Add success. I1122 05:50:56.449704 3798194 net.go:778] primary dev: ETH0 I1122 05:50:56.449718 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:50:56.449732 3798194 net.go:659] Add eni metrics success, count: 7 E1122 05:50:57.451071 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 05:50:57.451085 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:50:57.451319 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:50:57.451329 3798194 disk_worker.go:723] disk space is not compliant W1122 05:50:57.451332 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:50:57.460047 3798194 disk_worker.go:509] system disk:vda1 I1122 05:50:57.460089 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 05:50:58.448070 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 05:50:58.448085 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 05:50:59.448739 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 05:50:59.449813 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:50:59.449864 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:50:59.449883 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:50:59.471434 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:51:02.480133 3798194 disk_info.go:125] begin check local disk info of client I1122 05:51:02.484018 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:51:02.484028 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040dd40 0xc00040dd80] E1122 05:51:06.412771 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:51:06.412795 3798194 memory.go:184] no items to output this cycle I1122 05:51:06.413157 3798194 cpu.go:275] no items to output this cycle E1122 05:51:16.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:51:16.413267 3798194 memory.go:184] no items to output this cycle I1122 05:51:16.413417 3798194 cpu.go:275] no items to output this cycle I1122 05:51:17.640389 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:51:17.640397 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:51:26.411221 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:51:26.411255 3798194 memory.go:191] Add success. I1122 05:51:26.411616 3798194 cpu.go:282] Add success. I1122 05:51:26.442205 3798194 net.go:642] Add success. I1122 05:51:26.454447 3798194 net.go:778] primary dev: ETH0 I1122 05:51:26.454472 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:51:26.454488 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:51:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:51:29.450120 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:51:29.450155 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:51:36.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:51:36.412255 3798194 memory.go:184] no items to output this cycle I1122 05:51:36.412396 3798194 cpu.go:275] no items to output this cycle E1122 05:51:46.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:51:46.413250 3798194 memory.go:184] no items to output this cycle I1122 05:51:46.413402 3798194 cpu.go:275] no items to output this cycle E1122 05:51:56.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:51:56.412272 3798194 memory.go:191] Add success. W1122 05:51:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:51:56.412317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:51:56.412321 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:51:56.416375 3798194 cpu.go:282] Add success. I1122 05:51:56.441759 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:51:56.442274 3798194 net.go:642] Add success. I1122 05:51:56.443744 3798194 net.go:778] primary dev: ETH0 I1122 05:51:56.443759 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:51:56.443771 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:51:57.451001 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:51:57.451199 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:51:57.451257 3798194 disk_worker.go:723] disk space is not compliant W1122 05:51:57.451260 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:51:57.460012 3798194 disk_worker.go:509] system disk:vda1 I1122 05:51:57.460058 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:51:58.446007 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:51:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:51:59.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:51:59.450123 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:51:59.471638 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:52:02.484143 3798194 disk_info.go:125] begin check local disk info of client I1122 05:52:02.488030 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:52:02.488039 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000372f80 0xc000372fc0] E1122 05:52:06.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:52:06.411972 3798194 memory.go:184] no items to output this cycle I1122 05:52:06.412128 3798194 cpu.go:275] no items to output this cycle E1122 05:52:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:52:16.411981 3798194 memory.go:184] no items to output this cycle I1122 05:52:16.412141 3798194 cpu.go:275] no items to output this cycle E1122 05:52:26.413250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:52:26.413283 3798194 memory.go:191] Add success. I1122 05:52:26.417540 3798194 cpu.go:282] Add success. I1122 05:52:26.442084 3798194 net.go:642] Add success. I1122 05:52:26.443619 3798194 net.go:778] primary dev: ETH0 I1122 05:52:26.443633 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:52:26.443647 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:52:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:52:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:52:29.450145 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:52:36.411945 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:52:36.411967 3798194 memory.go:184] no items to output this cycle I1122 05:52:36.412105 3798194 cpu.go:275] no items to output this cycle E1122 05:52:46.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:52:46.412272 3798194 memory.go:184] no items to output this cycle I1122 05:52:46.412418 3798194 cpu.go:275] no items to output this cycle E1122 05:52:56.412246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:52:56.412278 3798194 memory.go:191] Add success. W1122 05:52:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:52:56.412323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:52:56.412326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:52:56.416642 3798194 cpu.go:282] Add success. I1122 05:52:56.440958 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:52:56.441550 3798194 net.go:642] Add success. I1122 05:52:56.442965 3798194 net.go:778] primary dev: ETH0 I1122 05:52:56.442979 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:52:56.442991 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:52:57.454570 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:52:57.454735 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:52:57.454791 3798194 disk_worker.go:723] disk space is not compliant W1122 05:52:57.454795 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:52:57.469144 3798194 disk_worker.go:509] system disk:vda1 I1122 05:52:57.469189 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:52:58.446194 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:52:59.450198 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:52:59.450278 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:52:59.450304 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:52:59.470724 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:53:02.488905 3798194 disk_info.go:125] begin check local disk info of client I1122 05:53:02.492800 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:53:02.492813 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e9180 0xc0003e91c0] E1122 05:53:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:53:06.412257 3798194 memory.go:184] no items to output this cycle I1122 05:53:06.412400 3798194 cpu.go:275] no items to output this cycle E1122 05:53:16.413454 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:53:16.413477 3798194 memory.go:184] no items to output this cycle I1122 05:53:16.413622 3798194 cpu.go:275] no items to output this cycle E1122 05:53:26.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:53:26.413261 3798194 memory.go:191] Add success. I1122 05:53:26.417639 3798194 cpu.go:282] Add success. I1122 05:53:26.442426 3798194 net.go:642] Add success. I1122 05:53:26.459831 3798194 net.go:778] primary dev: ETH0 I1122 05:53:26.459849 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:53:26.459864 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:53:29.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:53:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:53:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:53:36.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:53:36.412262 3798194 memory.go:184] no items to output this cycle I1122 05:53:36.412405 3798194 cpu.go:275] no items to output this cycle E1122 05:53:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:53:46.411973 3798194 memory.go:184] no items to output this cycle I1122 05:53:46.412128 3798194 cpu.go:275] no items to output this cycle E1122 05:53:56.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:53:56.413273 3798194 memory.go:191] Add success. W1122 05:53:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:53:56.413316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:53:56.413320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:53:56.417606 3798194 cpu.go:282] Add success. I1122 05:53:56.440946 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:53:56.441551 3798194 net.go:642] Add success. I1122 05:53:56.443260 3798194 net.go:778] primary dev: ETH0 I1122 05:53:56.443277 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:53:56.443289 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:53:57.445970 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:53:57.446193 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:53:57.446274 3798194 disk_worker.go:723] disk space is not compliant W1122 05:53:57.446277 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:53:57.451410 3798194 disk_worker.go:509] system disk:vda1 I1122 05:53:57.451463 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:53:58.446402 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:53:59.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:53:59.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:53:59.450498 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:53:59.471022 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:54:02.492916 3798194 disk_info.go:125] begin check local disk info of client I1122 05:54:02.496788 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:54:02.496797 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000483700 0xc000483740] E1122 05:54:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:54:06.411983 3798194 memory.go:184] no items to output this cycle I1122 05:54:06.412133 3798194 cpu.go:275] no items to output this cycle E1122 05:54:16.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:54:16.411988 3798194 memory.go:184] no items to output this cycle I1122 05:54:16.412154 3798194 cpu.go:275] no items to output this cycle I1122 05:54:17.640567 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:54:17.640575 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:54:26.412728 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:54:26.412769 3798194 memory.go:191] Add success. I1122 05:54:26.417025 3798194 cpu.go:282] Add success. I1122 05:54:26.441606 3798194 net.go:642] Add success. I1122 05:54:26.443214 3798194 net.go:778] primary dev: ETH0 I1122 05:54:26.443228 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:54:26.443241 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:54:29.450192 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:54:29.450271 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:54:29.450297 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:54:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:54:36.413265 3798194 memory.go:184] no items to output this cycle I1122 05:54:36.413407 3798194 cpu.go:275] no items to output this cycle E1122 05:54:46.411251 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:54:46.411274 3798194 memory.go:184] no items to output this cycle I1122 05:54:46.411423 3798194 cpu.go:275] no items to output this cycle E1122 05:54:56.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:54:56.412885 3798194 memory.go:191] Add success. W1122 05:54:56.412916 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:54:56.412928 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:54:56.412931 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:54:56.417139 3798194 cpu.go:282] Add success. I1122 05:54:56.441415 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:54:56.441972 3798194 net.go:642] Add success. I1122 05:54:56.443526 3798194 net.go:778] primary dev: ETH0 I1122 05:54:56.443540 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:54:56.443552 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:54:57.450685 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:54:57.450849 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:54:57.450904 3798194 disk_worker.go:723] disk space is not compliant W1122 05:54:57.450907 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:54:57.460678 3798194 disk_worker.go:509] system disk:vda1 I1122 05:54:57.460721 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:54:58.445997 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:54:59.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:54:59.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:54:59.450500 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:54:59.471025 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:55:02.496890 3798194 disk_info.go:125] begin check local disk info of client I1122 05:55:02.500753 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:55:02.500761 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d83c0 0xc0003d8400] E1122 05:55:06.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:55:06.412247 3798194 memory.go:184] no items to output this cycle I1122 05:55:06.412396 3798194 cpu.go:275] no items to output this cycle E1122 05:55:16.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:55:16.413250 3798194 memory.go:184] no items to output this cycle I1122 05:55:16.413393 3798194 cpu.go:275] no items to output this cycle E1122 05:55:26.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:55:26.412271 3798194 memory.go:191] Add success. I1122 05:55:26.416672 3798194 cpu.go:282] Add success. I1122 05:55:26.441291 3798194 net.go:642] Add success. I1122 05:55:26.442751 3798194 net.go:778] primary dev: ETH0 I1122 05:55:26.442766 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:55:26.442778 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:55:29.450403 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:55:29.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:55:29.450513 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:55:36.412258 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:55:36.412281 3798194 memory.go:184] no items to output this cycle I1122 05:55:36.412430 3798194 cpu.go:275] no items to output this cycle E1122 05:55:46.412224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:55:46.412247 3798194 memory.go:184] no items to output this cycle I1122 05:55:46.412387 3798194 cpu.go:275] no items to output this cycle E1122 05:55:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:55:56.413270 3798194 memory.go:191] Add success. W1122 05:55:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:55:56.413309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:55:56.413313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:55:56.417893 3798194 cpu.go:282] Add success. I1122 05:55:56.443823 3798194 event_worker.go:152] Polling the log file for events... I1122 05:55:56.444028 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:55:56.444533 3798194 net.go:642] Add success. I1122 05:55:56.446069 3798194 net.go:778] primary dev: ETH0 I1122 05:55:56.446083 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:55:56.446097 3798194 net.go:659] Add eni metrics success, count: 7 E1122 05:55:57.450332 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 05:55:57.450346 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:55:57.450507 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:55:57.450558 3798194 disk_worker.go:723] disk space is not compliant W1122 05:55:57.450561 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:55:57.458969 3798194 disk_worker.go:509] system disk:vda1 I1122 05:55:57.459011 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 05:55:58.448454 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 05:55:58.448470 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:55:59.449729 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:55:59.449780 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:55:59.449798 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:55:59.449910 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 05:55:59.471343 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:56:02.501661 3798194 disk_info.go:125] begin check local disk info of client I1122 05:56:02.505532 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:56:02.505540 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000275b40 0xc000275b80] E1122 05:56:06.412376 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:56:06.412405 3798194 memory.go:184] no items to output this cycle I1122 05:56:06.412614 3798194 cpu.go:275] no items to output this cycle E1122 05:56:16.412995 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:56:16.413019 3798194 memory.go:184] no items to output this cycle I1122 05:56:16.413181 3798194 cpu.go:275] no items to output this cycle E1122 05:56:26.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:56:26.412273 3798194 memory.go:191] Add success. I1122 05:56:26.416784 3798194 cpu.go:282] Add success. I1122 05:56:26.441289 3798194 net.go:642] Add success. I1122 05:56:26.442865 3798194 net.go:778] primary dev: ETH0 I1122 05:56:26.442879 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:56:26.442893 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:56:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:56:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:56:29.450144 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:56:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:56:36.411976 3798194 memory.go:184] no items to output this cycle I1122 05:56:36.412130 3798194 cpu.go:275] no items to output this cycle E1122 05:56:46.411977 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:56:46.412010 3798194 memory.go:184] no items to output this cycle I1122 05:56:46.412168 3798194 cpu.go:275] no items to output this cycle E1122 05:56:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:56:56.411954 3798194 memory.go:191] Add success. W1122 05:56:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:56:56.412003 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:56:56.412006 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:56:56.412388 3798194 cpu.go:282] Add success. I1122 05:56:56.440982 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:56:56.441602 3798194 net.go:642] Add success. I1122 05:56:56.446012 3798194 net.go:778] primary dev: ETH0 I1122 05:56:56.446029 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:56:56.446043 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:56:57.451284 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:56:57.451511 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:56:57.451535 3798194 disk_worker.go:723] disk space is not compliant W1122 05:56:57.451538 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:56:57.464250 3798194 disk_worker.go:509] system disk:vda1 I1122 05:56:57.464298 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:56:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:56:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:56:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:56:59.450147 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:56:59.470697 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:57:02.506204 3798194 disk_info.go:125] begin check local disk info of client I1122 05:57:02.510052 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:57:02.510062 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030f400 0xc00030f440] E1122 05:57:06.413107 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:57:06.413147 3798194 memory.go:184] no items to output this cycle I1122 05:57:06.413293 3798194 cpu.go:275] no items to output this cycle E1122 05:57:16.412343 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:57:16.412368 3798194 memory.go:184] no items to output this cycle I1122 05:57:16.412516 3798194 cpu.go:275] no items to output this cycle I1122 05:57:17.640744 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 05:57:17.640754 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 05:57:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:57:26.413269 3798194 memory.go:191] Add success. I1122 05:57:26.417849 3798194 cpu.go:282] Add success. I1122 05:57:26.441379 3798194 net.go:642] Add success. I1122 05:57:26.442785 3798194 net.go:778] primary dev: ETH0 I1122 05:57:26.442799 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:57:26.442811 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:57:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:57:29.450119 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:57:29.450148 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:57:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:57:36.411976 3798194 memory.go:184] no items to output this cycle I1122 05:57:36.412133 3798194 cpu.go:275] no items to output this cycle E1122 05:57:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:57:46.411983 3798194 memory.go:184] no items to output this cycle I1122 05:57:46.412149 3798194 cpu.go:275] no items to output this cycle E1122 05:57:56.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:57:56.413279 3798194 memory.go:191] Add success. W1122 05:57:56.413315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:57:56.413327 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:57:56.413330 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:57:56.417403 3798194 cpu.go:282] Add success. I1122 05:57:56.441792 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:57:56.442357 3798194 net.go:642] Add success. I1122 05:57:56.443731 3798194 net.go:778] primary dev: ETH0 I1122 05:57:56.443745 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:57:56.443758 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:57:57.451749 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:57:57.451971 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:57:57.451984 3798194 disk_worker.go:723] disk space is not compliant W1122 05:57:57.451987 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:57:57.460843 3798194 disk_worker.go:509] system disk:vda1 I1122 05:57:57.460884 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:57:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:57:59.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:57:59.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:57:59.450510 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:57:59.471041 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:58:02.510156 3798194 disk_info.go:125] begin check local disk info of client I1122 05:58:02.514009 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:58:02.514019 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000461280 0xc0004612c0] E1122 05:58:06.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:58:06.411970 3798194 memory.go:184] no items to output this cycle I1122 05:58:06.412327 3798194 cpu.go:275] no items to output this cycle E1122 05:58:16.412847 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:58:16.412870 3798194 memory.go:184] no items to output this cycle I1122 05:58:16.413012 3798194 cpu.go:275] no items to output this cycle E1122 05:58:26.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:58:26.413266 3798194 memory.go:191] Add success. I1122 05:58:26.417321 3798194 cpu.go:282] Add success. I1122 05:58:26.441863 3798194 net.go:642] Add success. I1122 05:58:26.443269 3798194 net.go:778] primary dev: ETH0 I1122 05:58:26.443283 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:58:26.443297 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:58:29.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:58:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:58:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:58:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:58:36.413258 3798194 memory.go:184] no items to output this cycle I1122 05:58:36.413419 3798194 cpu.go:275] no items to output this cycle E1122 05:58:46.411706 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:58:46.411730 3798194 memory.go:184] no items to output this cycle I1122 05:58:46.411884 3798194 cpu.go:275] no items to output this cycle E1122 05:58:56.413575 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:58:56.413609 3798194 memory.go:191] Add success. W1122 05:58:56.413644 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:58:56.413656 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:58:56.413659 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:58:56.418012 3798194 cpu.go:282] Add success. I1122 05:58:56.441355 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:58:56.441834 3798194 net.go:642] Add success. I1122 05:58:56.443362 3798194 net.go:778] primary dev: ETH0 I1122 05:58:56.443378 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:58:56.443392 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:58:57.445921 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:58:57.446224 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:58:57.446237 3798194 disk_worker.go:723] disk space is not compliant W1122 05:58:57.446240 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:58:57.451338 3798194 disk_worker.go:509] system disk:vda1 I1122 05:58:57.451385 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:58:58.446366 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:58:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:58:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:58:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:58:59.471652 3798194 disk_local_worker.go:436] Get disk info: [] I1122 05:59:02.514138 3798194 disk_info.go:125] begin check local disk info of client I1122 05:59:02.518052 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 05:59:02.518061 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000365000 0xc000365040] E1122 05:59:06.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:59:06.412261 3798194 memory.go:184] no items to output this cycle I1122 05:59:06.412402 3798194 cpu.go:275] no items to output this cycle E1122 05:59:16.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:59:16.413252 3798194 memory.go:184] no items to output this cycle I1122 05:59:16.413395 3798194 cpu.go:275] no items to output this cycle E1122 05:59:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:59:26.411950 3798194 memory.go:191] Add success. I1122 05:59:26.412298 3798194 cpu.go:282] Add success. I1122 05:59:26.441742 3798194 net.go:642] Add success. I1122 05:59:26.446632 3798194 net.go:778] primary dev: ETH0 I1122 05:59:26.446648 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:59:26.446661 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:59:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:59:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:59:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 05:59:36.413160 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:59:36.413183 3798194 memory.go:184] no items to output this cycle I1122 05:59:36.413454 3798194 cpu.go:275] no items to output this cycle E1122 05:59:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:59:46.411980 3798194 memory.go:184] no items to output this cycle I1122 05:59:46.412140 3798194 cpu.go:275] no items to output this cycle E1122 05:59:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 05:59:56.411949 3798194 memory.go:191] Add success. W1122 05:59:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 05:59:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 05:59:56.411991 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 05:59:56.412366 3798194 cpu.go:282] Add success. I1122 05:59:56.441827 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 05:59:56.442420 3798194 net.go:642] Add success. I1122 05:59:56.446571 3798194 net.go:778] primary dev: ETH0 I1122 05:59:56.446586 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 05:59:56.446599 3798194 net.go:659] Add eni metrics success, count: 7 I1122 05:59:57.450631 3798194 custom_config.go:106] query custom config with name: gpu W1122 05:59:57.450831 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 05:59:57.450842 3798194 disk_worker.go:723] disk space is not compliant W1122 05:59:57.450846 3798194 disk_worker.go:743] disk inode is not compliant I1122 05:59:57.459242 3798194 disk_worker.go:509] system disk:vda1 I1122 05:59:57.459285 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 05:59:58.445995 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 05:59:59.450098 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 05:59:59.450183 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 05:59:59.450211 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 05:59:59.470677 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:00:02.518162 3798194 disk_info.go:125] begin check local disk info of client I1122 06:00:02.522072 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:00:02.522081 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00043b840 0xc00043b880] E1122 06:00:06.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:00:06.412882 3798194 memory.go:184] no items to output this cycle I1122 06:00:06.413028 3798194 cpu.go:275] no items to output this cycle E1122 06:00:16.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:00:16.413246 3798194 memory.go:184] no items to output this cycle I1122 06:00:16.413389 3798194 cpu.go:275] no items to output this cycle I1122 06:00:17.641214 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:00:17.641223 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:00:26.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:00:26.412273 3798194 memory.go:191] Add success. I1122 06:00:26.416182 3798194 cpu.go:282] Add success. I1122 06:00:26.441751 3798194 net.go:642] Add success. I1122 06:00:26.443250 3798194 net.go:778] primary dev: ETH0 I1122 06:00:26.443264 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:00:26.443279 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:00:29.450406 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:00:29.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:00:29.450513 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:00:36.413657 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:00:36.413681 3798194 memory.go:184] no items to output this cycle I1122 06:00:36.413823 3798194 cpu.go:275] no items to output this cycle E1122 06:00:46.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:00:46.412262 3798194 memory.go:184] no items to output this cycle I1122 06:00:46.412409 3798194 cpu.go:275] no items to output this cycle E1122 06:00:56.412708 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:00:56.412742 3798194 memory.go:191] Add success. W1122 06:00:56.412776 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:00:56.412787 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:00:56.412791 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:00:56.417048 3798194 cpu.go:282] Add success. I1122 06:00:56.446656 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:00:56.447221 3798194 net.go:642] Add success. I1122 06:00:56.448678 3798194 net.go:778] primary dev: ETH0 I1122 06:00:56.448695 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:00:56.448708 3798194 net.go:659] Add eni metrics success, count: 7 E1122 06:00:57.450588 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 06:00:57.450604 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:00:57.451061 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:00:57.451071 3798194 disk_worker.go:723] disk space is not compliant W1122 06:00:57.451074 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:00:57.460152 3798194 disk_worker.go:509] system disk:vda1 I1122 06:00:57.460192 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 06:00:58.448096 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 06:00:58.448127 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 06:00:59.448807 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 06:00:59.449882 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:00:59.449933 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:00:59.449952 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:00:59.471492 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:01:02.522187 3798194 disk_info.go:125] begin check local disk info of client I1122 06:01:02.525984 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:01:02.525992 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000481ac0 0xc000481b00] E1122 06:01:06.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:01:06.413257 3798194 memory.go:184] no items to output this cycle I1122 06:01:06.413398 3798194 cpu.go:275] no items to output this cycle E1122 06:01:16.412286 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:01:16.412318 3798194 memory.go:184] no items to output this cycle I1122 06:01:16.412550 3798194 cpu.go:275] no items to output this cycle E1122 06:01:26.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:01:26.413273 3798194 memory.go:191] Add success. I1122 06:01:26.417630 3798194 cpu.go:282] Add success. I1122 06:01:26.442059 3798194 net.go:642] Add success. I1122 06:01:26.443660 3798194 net.go:778] primary dev: ETH0 I1122 06:01:26.443675 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:01:26.443688 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:01:29.450208 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:01:29.450279 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:01:29.450307 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:01:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:01:36.411978 3798194 memory.go:184] no items to output this cycle I1122 06:01:36.412128 3798194 cpu.go:275] no items to output this cycle E1122 06:01:46.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:01:46.412260 3798194 memory.go:184] no items to output this cycle I1122 06:01:46.412408 3798194 cpu.go:275] no items to output this cycle E1122 06:01:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:01:56.411950 3798194 memory.go:191] Add success. W1122 06:01:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:01:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:01:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:01:56.412353 3798194 cpu.go:282] Add success. I1122 06:01:56.441817 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:01:56.442411 3798194 net.go:642] Add success. I1122 06:01:56.446249 3798194 net.go:778] primary dev: ETH0 I1122 06:01:56.446265 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:01:56.446277 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:01:57.450542 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:01:57.450699 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:01:57.450750 3798194 disk_worker.go:723] disk space is not compliant W1122 06:01:57.450753 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:01:57.459073 3798194 disk_worker.go:509] system disk:vda1 I1122 06:01:57.459128 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:01:58.445997 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:01:59.450007 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:01:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:01:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:01:59.471700 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:02:02.526136 3798194 disk_info.go:125] begin check local disk info of client I1122 06:02:02.530000 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:02:02.530008 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b8d00 0xc0003b8d40] E1122 06:02:06.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:02:06.413256 3798194 memory.go:184] no items to output this cycle I1122 06:02:06.413398 3798194 cpu.go:275] no items to output this cycle E1122 06:02:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:02:16.411984 3798194 memory.go:184] no items to output this cycle I1122 06:02:16.412139 3798194 cpu.go:275] no items to output this cycle E1122 06:02:26.413257 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:02:26.413304 3798194 memory.go:191] Add success. I1122 06:02:26.417587 3798194 cpu.go:282] Add success. I1122 06:02:26.441516 3798194 net.go:642] Add success. I1122 06:02:26.443070 3798194 net.go:778] primary dev: ETH0 I1122 06:02:26.443084 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:02:26.443096 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:02:29.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:02:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:02:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:02:36.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:02:36.413253 3798194 memory.go:184] no items to output this cycle I1122 06:02:36.413395 3798194 cpu.go:275] no items to output this cycle E1122 06:02:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:02:46.412253 3798194 memory.go:184] no items to output this cycle I1122 06:02:46.412405 3798194 cpu.go:275] no items to output this cycle E1122 06:02:56.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:02:56.412259 3798194 memory.go:191] Add success. W1122 06:02:56.412293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:02:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:02:56.412308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:02:56.416231 3798194 cpu.go:282] Add success. I1122 06:02:56.441641 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:02:56.442231 3798194 net.go:642] Add success. I1122 06:02:56.443624 3798194 net.go:778] primary dev: ETH0 I1122 06:02:56.443639 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:02:56.443650 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:02:57.450994 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:02:57.451224 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:02:57.451235 3798194 disk_worker.go:723] disk space is not compliant W1122 06:02:57.451238 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:02:57.459963 3798194 disk_worker.go:509] system disk:vda1 I1122 06:02:57.460005 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:02:58.446587 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:02:59.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:02:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:02:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:02:59.471673 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:03:02.530811 3798194 disk_info.go:125] begin check local disk info of client I1122 06:03:02.534752 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:03:02.534762 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000326c40 0xc000326c80] E1122 06:03:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:03:06.411983 3798194 memory.go:184] no items to output this cycle I1122 06:03:06.412140 3798194 cpu.go:275] no items to output this cycle E1122 06:03:16.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:03:16.412250 3798194 memory.go:184] no items to output this cycle I1122 06:03:16.412405 3798194 cpu.go:275] no items to output this cycle I1122 06:03:17.641384 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:03:17.641395 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:03:26.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:03:26.412266 3798194 memory.go:191] Add success. I1122 06:03:26.416348 3798194 cpu.go:282] Add success. I1122 06:03:26.441884 3798194 net.go:642] Add success. I1122 06:03:26.443492 3798194 net.go:778] primary dev: ETH0 I1122 06:03:26.443507 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:03:26.443527 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:03:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:03:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:03:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:03:36.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:03:36.412265 3798194 memory.go:184] no items to output this cycle I1122 06:03:36.412409 3798194 cpu.go:275] no items to output this cycle E1122 06:03:46.411242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:03:46.411270 3798194 memory.go:184] no items to output this cycle I1122 06:03:46.411420 3798194 cpu.go:275] no items to output this cycle E1122 06:03:56.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:03:56.413272 3798194 memory.go:191] Add success. W1122 06:03:56.413376 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:03:56.413389 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:03:56.413392 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:03:56.417945 3798194 cpu.go:282] Add success. I1122 06:03:56.441419 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:03:56.441888 3798194 net.go:642] Add success. I1122 06:03:56.443370 3798194 net.go:778] primary dev: ETH0 I1122 06:03:56.443384 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:03:56.443397 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:03:57.446488 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:03:57.446746 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:03:57.446760 3798194 disk_worker.go:723] disk space is not compliant W1122 06:03:57.446763 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:03:57.449554 3798194 disk_worker.go:509] system disk:vda1 I1122 06:03:57.449593 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:03:58.446373 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:03:59.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:03:59.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:03:59.450508 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:03:59.471062 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:04:02.534862 3798194 disk_info.go:125] begin check local disk info of client I1122 06:04:02.538794 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:04:02.538804 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e480 0xc00030e4c0] E1122 06:04:06.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:04:06.411972 3798194 memory.go:184] no items to output this cycle I1122 06:04:06.412131 3798194 cpu.go:275] no items to output this cycle E1122 06:04:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:04:16.411976 3798194 memory.go:184] no items to output this cycle I1122 06:04:16.412136 3798194 cpu.go:275] no items to output this cycle E1122 06:04:26.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:04:26.411963 3798194 memory.go:191] Add success. I1122 06:04:26.412300 3798194 cpu.go:282] Add success. I1122 06:04:26.441881 3798194 net.go:642] Add success. I1122 06:04:26.446032 3798194 net.go:778] primary dev: ETH0 I1122 06:04:26.446047 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:04:26.446060 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:04:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:04:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:04:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:04:36.412909 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:04:36.412931 3798194 memory.go:184] no items to output this cycle I1122 06:04:36.413073 3798194 cpu.go:275] no items to output this cycle E1122 06:04:46.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:04:46.412257 3798194 memory.go:184] no items to output this cycle I1122 06:04:46.412397 3798194 cpu.go:275] no items to output this cycle E1122 06:04:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:04:56.411949 3798194 memory.go:191] Add success. W1122 06:04:56.411981 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:04:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:04:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:04:56.412338 3798194 cpu.go:282] Add success. I1122 06:04:56.441786 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:04:56.442299 3798194 net.go:642] Add success. I1122 06:04:56.446052 3798194 net.go:778] primary dev: ETH0 I1122 06:04:56.446068 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:04:56.446082 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:04:57.450952 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:04:57.451132 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:04:57.451192 3798194 disk_worker.go:723] disk space is not compliant W1122 06:04:57.451195 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:04:57.461294 3798194 disk_worker.go:509] system disk:vda1 I1122 06:04:57.461347 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:04:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:04:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:04:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:04:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:04:59.470681 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:05:02.538899 3798194 disk_info.go:125] begin check local disk info of client I1122 06:05:02.542745 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:05:02.542754 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046a100 0xc00046a140] E1122 06:05:06.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:05:06.412348 3798194 memory.go:184] no items to output this cycle I1122 06:05:06.412675 3798194 cpu.go:275] no items to output this cycle E1122 06:05:16.412894 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:05:16.412916 3798194 memory.go:184] no items to output this cycle I1122 06:05:16.413071 3798194 cpu.go:275] no items to output this cycle E1122 06:05:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:05:26.411956 3798194 memory.go:191] Add success. I1122 06:05:26.412320 3798194 cpu.go:282] Add success. I1122 06:05:26.441897 3798194 net.go:642] Add success. I1122 06:05:26.445990 3798194 net.go:778] primary dev: ETH0 I1122 06:05:26.446008 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:05:26.446022 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:05:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:05:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:05:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:05:36.412946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:05:36.412968 3798194 memory.go:184] no items to output this cycle I1122 06:05:36.413140 3798194 cpu.go:275] no items to output this cycle E1122 06:05:46.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:05:46.411967 3798194 memory.go:184] no items to output this cycle I1122 06:05:46.412129 3798194 cpu.go:275] no items to output this cycle E1122 06:05:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:05:56.411950 3798194 memory.go:191] Add success. W1122 06:05:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:05:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:05:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:05:56.412350 3798194 cpu.go:282] Add success. I1122 06:05:56.441563 3798194 event_worker.go:152] Polling the log file for events... I1122 06:05:56.444807 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:05:56.445299 3798194 net.go:642] Add success. I1122 06:05:56.449234 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795296956, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 06:05:56.449349 3798194 net.go:778] primary dev: ETH0 I1122 06:05:56.449362 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:05:56.449376 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:05:56.460958 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 06:05:57.445579 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 06:05:57.445597 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:05:57.445874 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:05:57.445885 3798194 disk_worker.go:723] disk space is not compliant W1122 06:05:57.445888 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:05:57.450857 3798194 disk_worker.go:509] system disk:vda1 I1122 06:05:57.450897 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 06:05:58.448103 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 06:05:58.448133 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 06:05:59.448661 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 06:05:59.449732 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:05:59.449784 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:05:59.449813 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:05:59.471355 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:06:02.542858 3798194 disk_info.go:125] begin check local disk info of client I1122 06:06:02.546778 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:06:02.546787 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000371740 0xc000371780] E1122 06:06:06.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:06:06.413254 3798194 memory.go:184] no items to output this cycle I1122 06:06:06.413399 3798194 cpu.go:275] no items to output this cycle E1122 06:06:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:06:16.411972 3798194 memory.go:184] no items to output this cycle I1122 06:06:16.412130 3798194 cpu.go:275] no items to output this cycle I1122 06:06:17.641560 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:06:17.641568 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:06:26.411225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:06:26.411257 3798194 memory.go:191] Add success. I1122 06:06:26.411581 3798194 cpu.go:282] Add success. I1122 06:06:26.442217 3798194 net.go:642] Add success. I1122 06:06:26.446567 3798194 net.go:778] primary dev: ETH0 I1122 06:06:26.446588 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:06:26.446691 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:06:29.449996 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:06:29.450079 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:06:29.450120 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:06:36.413167 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:06:36.413190 3798194 memory.go:184] no items to output this cycle I1122 06:06:36.413331 3798194 cpu.go:275] no items to output this cycle E1122 06:06:46.412500 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:06:46.412524 3798194 memory.go:184] no items to output this cycle I1122 06:06:46.412670 3798194 cpu.go:275] no items to output this cycle E1122 06:06:56.412522 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:06:56.412555 3798194 memory.go:191] Add success. W1122 06:06:56.412586 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:06:56.412597 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:06:56.412601 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:06:56.416528 3798194 cpu.go:282] Add success. I1122 06:06:56.441921 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:06:56.442475 3798194 net.go:642] Add success. I1122 06:06:56.444024 3798194 net.go:778] primary dev: ETH0 I1122 06:06:56.444038 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:06:56.444050 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:06:57.451057 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:06:57.451299 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:06:57.451310 3798194 disk_worker.go:723] disk space is not compliant W1122 06:06:57.451313 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:06:57.459720 3798194 disk_worker.go:509] system disk:vda1 I1122 06:06:57.459763 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:06:58.446021 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:06:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:06:59.450116 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:06:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:06:59.471666 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:07:02.546896 3798194 disk_info.go:125] begin check local disk info of client I1122 06:07:02.550746 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:07:02.550754 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ebc00 0xc0003ebc40] E1122 06:07:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:07:06.413266 3798194 memory.go:184] no items to output this cycle I1122 06:07:06.413443 3798194 cpu.go:275] no items to output this cycle E1122 06:07:16.412834 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:07:16.412858 3798194 memory.go:184] no items to output this cycle I1122 06:07:16.413012 3798194 cpu.go:275] no items to output this cycle E1122 06:07:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:07:26.411956 3798194 memory.go:191] Add success. I1122 06:07:26.412318 3798194 cpu.go:282] Add success. I1122 06:07:26.441844 3798194 net.go:642] Add success. I1122 06:07:26.445853 3798194 net.go:778] primary dev: ETH0 I1122 06:07:26.445869 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:07:26.445883 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:07:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:07:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:07:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:07:36.412958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:07:36.412980 3798194 memory.go:184] no items to output this cycle I1122 06:07:36.413137 3798194 cpu.go:275] no items to output this cycle E1122 06:07:46.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:07:46.413252 3798194 memory.go:184] no items to output this cycle I1122 06:07:46.413408 3798194 cpu.go:275] no items to output this cycle E1122 06:07:56.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:07:56.412890 3798194 memory.go:191] Add success. W1122 06:07:56.412923 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:07:56.412935 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:07:56.412938 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:07:56.417350 3798194 cpu.go:282] Add success. I1122 06:07:56.441885 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:07:56.442844 3798194 net.go:642] Add success. I1122 06:07:56.444414 3798194 net.go:778] primary dev: ETH0 I1122 06:07:56.444431 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:07:56.444448 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:07:57.451551 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:07:57.451733 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:07:57.451789 3798194 disk_worker.go:723] disk space is not compliant W1122 06:07:57.451792 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:07:57.460303 3798194 disk_worker.go:509] system disk:vda1 I1122 06:07:57.460345 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:07:58.446374 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:07:59.450386 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:07:59.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:07:59.450502 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:07:59.471040 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:08:02.550857 3798194 disk_info.go:125] begin check local disk info of client I1122 06:08:02.554700 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:08:02.554709 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004bfa40 0xc0004bfa80] E1122 06:08:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:08:06.411975 3798194 memory.go:184] no items to output this cycle I1122 06:08:06.412152 3798194 cpu.go:275] no items to output this cycle E1122 06:08:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:08:16.411984 3798194 memory.go:184] no items to output this cycle I1122 06:08:16.412142 3798194 cpu.go:275] no items to output this cycle E1122 06:08:26.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:08:26.412264 3798194 memory.go:191] Add success. I1122 06:08:26.417159 3798194 cpu.go:282] Add success. I1122 06:08:26.441668 3798194 net.go:642] Add success. I1122 06:08:26.443216 3798194 net.go:778] primary dev: ETH0 I1122 06:08:26.443234 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:08:26.443247 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:08:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:08:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:08:29.450152 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:08:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:08:36.413258 3798194 memory.go:184] no items to output this cycle I1122 06:08:36.413407 3798194 cpu.go:275] no items to output this cycle E1122 06:08:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:08:46.412253 3798194 memory.go:184] no items to output this cycle I1122 06:08:46.412402 3798194 cpu.go:275] no items to output this cycle E1122 06:08:56.411108 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:08:56.411156 3798194 memory.go:191] Add success. W1122 06:08:56.411185 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:08:56.411197 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:08:56.411201 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:08:56.415648 3798194 cpu.go:282] Add success. I1122 06:08:56.440946 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:08:56.441538 3798194 net.go:642] Add success. I1122 06:08:56.443006 3798194 net.go:778] primary dev: ETH0 I1122 06:08:56.443021 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:08:56.443033 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:08:57.451391 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:08:57.451549 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:08:57.451599 3798194 disk_worker.go:723] disk space is not compliant W1122 06:08:57.451603 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:08:57.459959 3798194 disk_worker.go:509] system disk:vda1 I1122 06:08:57.460002 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:08:58.446374 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:08:59.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:08:59.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:08:59.450500 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:08:59.471038 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:09:02.554806 3798194 disk_info.go:125] begin check local disk info of client I1122 06:09:02.558683 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:09:02.558692 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000438b40 0xc000438b80] E1122 06:09:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:09:06.412255 3798194 memory.go:184] no items to output this cycle I1122 06:09:06.412614 3798194 cpu.go:275] no items to output this cycle E1122 06:09:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:09:16.411983 3798194 memory.go:184] no items to output this cycle I1122 06:09:16.412140 3798194 cpu.go:275] no items to output this cycle I1122 06:09:17.641732 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:09:17.641740 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:09:26.411196 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:09:26.411235 3798194 memory.go:191] Add success. I1122 06:09:26.411591 3798194 cpu.go:282] Add success. I1122 06:09:26.442239 3798194 net.go:642] Add success. I1122 06:09:26.446673 3798194 net.go:778] primary dev: ETH0 I1122 06:09:26.446691 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:09:26.446706 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:09:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:09:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:09:29.450154 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:09:36.412812 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:09:36.412834 3798194 memory.go:184] no items to output this cycle I1122 06:09:36.412979 3798194 cpu.go:275] no items to output this cycle E1122 06:09:46.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:09:46.411969 3798194 memory.go:184] no items to output this cycle I1122 06:09:46.412132 3798194 cpu.go:275] no items to output this cycle E1122 06:09:56.412224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:09:56.412260 3798194 memory.go:191] Add success. W1122 06:09:56.412286 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:09:56.412298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:09:56.412301 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:09:56.416715 3798194 cpu.go:282] Add success. I1122 06:09:56.441154 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:09:56.441637 3798194 net.go:642] Add success. I1122 06:09:56.443153 3798194 net.go:778] primary dev: ETH0 I1122 06:09:56.443168 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:09:56.443182 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:09:57.450341 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:09:57.450514 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:09:57.450565 3798194 disk_worker.go:723] disk space is not compliant W1122 06:09:57.450568 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:09:57.458902 3798194 disk_worker.go:509] system disk:vda1 I1122 06:09:57.458944 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:09:58.446067 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:09:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:09:59.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:09:59.450500 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:09:59.471012 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:10:02.559572 3798194 disk_info.go:125] begin check local disk info of client I1122 06:10:02.563404 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:10:02.563413 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000373000 0xc000373040] E1122 06:10:06.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:10:06.412265 3798194 memory.go:184] no items to output this cycle I1122 06:10:06.412404 3798194 cpu.go:275] no items to output this cycle E1122 06:10:16.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:10:16.412258 3798194 memory.go:184] no items to output this cycle I1122 06:10:16.412419 3798194 cpu.go:275] no items to output this cycle E1122 06:10:26.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:10:26.412260 3798194 memory.go:191] Add success. I1122 06:10:26.416993 3798194 cpu.go:282] Add success. I1122 06:10:26.441526 3798194 net.go:642] Add success. I1122 06:10:26.443054 3798194 net.go:778] primary dev: ETH0 I1122 06:10:26.443071 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:10:26.443084 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:10:29.450383 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:10:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:10:29.450495 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:10:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:10:36.411979 3798194 memory.go:184] no items to output this cycle I1122 06:10:36.412331 3798194 cpu.go:275] no items to output this cycle E1122 06:10:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:10:46.412254 3798194 memory.go:184] no items to output this cycle I1122 06:10:46.412405 3798194 cpu.go:275] no items to output this cycle E1122 06:10:56.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:10:56.412260 3798194 memory.go:191] Add success. W1122 06:10:56.412293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:10:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:10:56.412308 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:10:56.416386 3798194 cpu.go:282] Add success. I1122 06:10:56.444673 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:10:56.445234 3798194 net.go:642] Add success. I1122 06:10:56.446787 3798194 net.go:778] primary dev: ETH0 I1122 06:10:56.446802 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:10:56.446814 3798194 net.go:659] Add eni metrics success, count: 7 E1122 06:10:57.451055 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 06:10:57.451068 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:10:57.451244 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:10:57.451299 3798194 disk_worker.go:723] disk space is not compliant W1122 06:10:57.451302 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:10:57.459739 3798194 disk_worker.go:509] system disk:vda1 I1122 06:10:57.459781 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 06:10:58.448684 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 06:10:58.448698 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 06:10:59.448621 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 06:10:59.449695 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:10:59.449749 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:10:59.449768 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:10:59.471317 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:11:02.563515 3798194 disk_info.go:125] begin check local disk info of client I1122 06:11:02.567377 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:11:02.567386 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000372d00 0xc000372d40] E1122 06:11:06.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:11:06.413259 3798194 memory.go:184] no items to output this cycle I1122 06:11:06.413416 3798194 cpu.go:275] no items to output this cycle E1122 06:11:16.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:11:16.412887 3798194 memory.go:184] no items to output this cycle I1122 06:11:16.413033 3798194 cpu.go:275] no items to output this cycle E1122 06:11:26.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:11:26.412886 3798194 memory.go:191] Add success. I1122 06:11:26.417367 3798194 cpu.go:282] Add success. I1122 06:11:26.441858 3798194 net.go:642] Add success. I1122 06:11:26.443333 3798194 net.go:778] primary dev: ETH0 I1122 06:11:26.443346 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:11:26.443360 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:11:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:11:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:11:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:11:36.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:11:36.411979 3798194 memory.go:184] no items to output this cycle I1122 06:11:36.412139 3798194 cpu.go:275] no items to output this cycle E1122 06:11:46.413338 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:11:46.413366 3798194 memory.go:184] no items to output this cycle I1122 06:11:46.413518 3798194 cpu.go:275] no items to output this cycle E1122 06:11:56.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:11:56.412896 3798194 memory.go:191] Add success. W1122 06:11:56.412929 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:11:56.412941 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:11:56.412946 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:11:56.417004 3798194 cpu.go:282] Add success. I1122 06:11:56.441368 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:11:56.441940 3798194 net.go:642] Add success. I1122 06:11:56.443377 3798194 net.go:778] primary dev: ETH0 I1122 06:11:56.443400 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:11:56.443413 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:11:57.450685 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:11:57.450897 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:11:57.450908 3798194 disk_worker.go:723] disk space is not compliant W1122 06:11:57.450911 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:11:57.459293 3798194 disk_worker.go:509] system disk:vda1 I1122 06:11:57.459346 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:11:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:11:59.450105 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:11:59.450194 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:11:59.450229 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:11:59.470692 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:12:02.567490 3798194 disk_info.go:125] begin check local disk info of client I1122 06:12:02.571388 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:12:02.571399 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000070000 0xc000070700] E1122 06:12:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:12:06.413256 3798194 memory.go:184] no items to output this cycle I1122 06:12:06.413399 3798194 cpu.go:275] no items to output this cycle E1122 06:12:16.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:12:16.411977 3798194 memory.go:184] no items to output this cycle I1122 06:12:16.412139 3798194 cpu.go:275] no items to output this cycle I1122 06:12:17.642218 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:12:17.642226 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:12:26.412309 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:12:26.412423 3798194 memory.go:191] Add success. I1122 06:12:26.439417 3798194 cpu.go:282] Add success. I1122 06:12:26.441876 3798194 net.go:642] Add success. I1122 06:12:26.452359 3798194 net.go:778] primary dev: ETH0 I1122 06:12:26.452384 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:12:26.452404 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:12:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:12:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:12:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:12:36.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:12:36.413271 3798194 memory.go:184] no items to output this cycle I1122 06:12:36.413413 3798194 cpu.go:275] no items to output this cycle E1122 06:12:46.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:12:46.412877 3798194 memory.go:184] no items to output this cycle I1122 06:12:46.413026 3798194 cpu.go:275] no items to output this cycle E1122 06:12:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:12:56.411953 3798194 memory.go:191] Add success. W1122 06:12:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:12:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:12:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:12:56.412340 3798194 cpu.go:282] Add success. I1122 06:12:56.441153 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:12:56.441726 3798194 net.go:642] Add success. I1122 06:12:56.445844 3798194 net.go:778] primary dev: ETH0 I1122 06:12:56.445858 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:12:56.445872 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:12:57.451089 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:12:57.451324 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:12:57.451335 3798194 disk_worker.go:723] disk space is not compliant W1122 06:12:57.451339 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:12:57.460119 3798194 disk_worker.go:509] system disk:vda1 I1122 06:12:57.460169 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:12:58.446381 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:12:59.450485 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:12:59.450561 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:12:59.450586 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:12:59.471076 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:13:02.571494 3798194 disk_info.go:125] begin check local disk info of client I1122 06:13:02.575327 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:13:02.575335 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fb1c0 0xc0003fb200] E1122 06:13:06.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:13:06.411978 3798194 memory.go:184] no items to output this cycle I1122 06:13:06.412130 3798194 cpu.go:275] no items to output this cycle E1122 06:13:16.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:13:16.412258 3798194 memory.go:184] no items to output this cycle I1122 06:13:16.412401 3798194 cpu.go:275] no items to output this cycle E1122 06:13:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:13:26.411961 3798194 memory.go:191] Add success. I1122 06:13:26.412346 3798194 cpu.go:282] Add success. I1122 06:13:26.441955 3798194 net.go:642] Add success. I1122 06:13:26.445678 3798194 net.go:778] primary dev: ETH0 I1122 06:13:26.445693 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:13:26.445707 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:13:29.450010 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:13:29.450086 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:13:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:13:36.412764 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:13:36.412785 3798194 memory.go:184] no items to output this cycle I1122 06:13:36.412939 3798194 cpu.go:275] no items to output this cycle E1122 06:13:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:13:46.412256 3798194 memory.go:184] no items to output this cycle I1122 06:13:46.412401 3798194 cpu.go:275] no items to output this cycle E1122 06:13:56.411266 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:13:56.411299 3798194 memory.go:191] Add success. W1122 06:13:56.411325 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:13:56.411337 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:13:56.411340 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:13:56.415831 3798194 cpu.go:282] Add success. I1122 06:13:56.441212 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:13:56.441712 3798194 net.go:642] Add success. I1122 06:13:56.443397 3798194 net.go:778] primary dev: ETH0 I1122 06:13:56.443413 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:13:56.443425 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:13:57.450911 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:13:57.451141 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:13:57.451153 3798194 disk_worker.go:723] disk space is not compliant W1122 06:13:57.451156 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:13:57.459879 3798194 disk_worker.go:509] system disk:vda1 I1122 06:13:57.459919 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:13:58.445945 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:13:59.450051 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:13:59.450141 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:13:59.450167 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:13:59.471693 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:14:02.575439 3798194 disk_info.go:125] begin check local disk info of client I1122 06:14:02.579250 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:14:02.579259 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046b640 0xc00046b680] E1122 06:14:06.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:14:06.411988 3798194 memory.go:184] no items to output this cycle I1122 06:14:06.412146 3798194 cpu.go:275] no items to output this cycle E1122 06:14:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:14:16.411980 3798194 memory.go:184] no items to output this cycle I1122 06:14:16.412136 3798194 cpu.go:275] no items to output this cycle E1122 06:14:26.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:14:26.412896 3798194 memory.go:191] Add success. I1122 06:14:26.417258 3798194 cpu.go:282] Add success. I1122 06:14:26.441961 3798194 net.go:642] Add success. I1122 06:14:26.443229 3798194 net.go:778] primary dev: ETH0 I1122 06:14:26.443243 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:14:26.443257 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:14:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:14:29.450127 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:14:29.450153 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:14:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:14:36.413261 3798194 memory.go:184] no items to output this cycle I1122 06:14:36.413411 3798194 cpu.go:275] no items to output this cycle E1122 06:14:46.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:14:46.413262 3798194 memory.go:184] no items to output this cycle I1122 06:14:46.413414 3798194 cpu.go:275] no items to output this cycle E1122 06:14:56.413046 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:14:56.413078 3798194 memory.go:191] Add success. W1122 06:14:56.413125 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:14:56.413140 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:14:56.413143 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:14:56.418150 3798194 cpu.go:282] Add success. I1122 06:14:56.441802 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:14:56.442293 3798194 net.go:642] Add success. I1122 06:14:56.443820 3798194 net.go:778] primary dev: ETH0 I1122 06:14:56.443834 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:14:56.443848 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:14:57.446463 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:14:57.446748 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:14:57.446759 3798194 disk_worker.go:723] disk space is not compliant W1122 06:14:57.446762 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:14:57.449519 3798194 disk_worker.go:509] system disk:vda1 I1122 06:14:57.449560 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:14:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:14:59.450485 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:14:59.450569 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:14:59.450595 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:14:59.471126 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:15:02.579361 3798194 disk_info.go:125] begin check local disk info of client I1122 06:15:02.583206 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:15:02.583216 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e440 0xc00030e480] E1122 06:15:06.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:15:06.411976 3798194 memory.go:184] no items to output this cycle I1122 06:15:06.412133 3798194 cpu.go:275] no items to output this cycle E1122 06:15:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:15:16.411974 3798194 memory.go:184] no items to output this cycle I1122 06:15:16.412139 3798194 cpu.go:275] no items to output this cycle I1122 06:15:17.642389 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:15:17.642398 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:15:26.411221 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:15:26.411256 3798194 memory.go:191] Add success. I1122 06:15:26.411587 3798194 cpu.go:282] Add success. I1122 06:15:26.441275 3798194 net.go:642] Add success. I1122 06:15:26.445042 3798194 net.go:778] primary dev: ETH0 I1122 06:15:26.445059 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:15:26.445072 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:15:29.449662 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:15:29.449749 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:15:29.449775 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:15:36.412835 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:15:36.412857 3798194 memory.go:184] no items to output this cycle I1122 06:15:36.412999 3798194 cpu.go:275] no items to output this cycle E1122 06:15:46.412796 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:15:46.412818 3798194 memory.go:184] no items to output this cycle I1122 06:15:46.412962 3798194 cpu.go:275] no items to output this cycle E1122 06:15:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:15:56.411956 3798194 memory.go:191] Add success. W1122 06:15:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:15:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:15:56.412001 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:15:56.412362 3798194 cpu.go:282] Add success. I1122 06:15:56.441696 3798194 event_worker.go:152] Polling the log file for events... I1122 06:15:56.444849 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:15:56.445432 3798194 net.go:642] Add success. I1122 06:15:56.449130 3798194 net.go:778] primary dev: ETH0 I1122 06:15:56.449145 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:15:56.449159 3798194 net.go:659] Add eni metrics success, count: 7 E1122 06:15:57.450410 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 06:15:57.450425 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:15:57.450581 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:15:57.450631 3798194 disk_worker.go:723] disk space is not compliant W1122 06:15:57.450634 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:15:57.458995 3798194 disk_worker.go:509] system disk:vda1 I1122 06:15:57.459040 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 06:15:58.448082 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 06:15:58.448097 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 06:15:59.448732 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 06:15:59.449807 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:15:59.449860 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:15:59.449878 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:15:59.471420 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:16:02.583310 3798194 disk_info.go:125] begin check local disk info of client I1122 06:16:02.587172 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:16:02.587184 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f9dc0 0xc0003f9e00] E1122 06:16:06.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:16:06.413387 3798194 memory.go:184] no items to output this cycle I1122 06:16:06.413677 3798194 cpu.go:275] no items to output this cycle E1122 06:16:16.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:16:16.411984 3798194 memory.go:184] no items to output this cycle I1122 06:16:16.412148 3798194 cpu.go:275] no items to output this cycle E1122 06:16:26.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:16:26.411944 3798194 memory.go:191] Add success. I1122 06:16:26.412300 3798194 cpu.go:282] Add success. I1122 06:16:26.441903 3798194 net.go:642] Add success. I1122 06:16:26.446506 3798194 net.go:778] primary dev: ETH0 I1122 06:16:26.446520 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:16:26.446534 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:16:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:16:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:16:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:16:36.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:16:36.413249 3798194 memory.go:184] no items to output this cycle I1122 06:16:36.413392 3798194 cpu.go:275] no items to output this cycle E1122 06:16:46.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:16:46.413255 3798194 memory.go:184] no items to output this cycle I1122 06:16:46.413402 3798194 cpu.go:275] no items to output this cycle E1122 06:16:56.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:16:56.412889 3798194 memory.go:191] Add success. W1122 06:16:56.412919 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:16:56.412931 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:16:56.412935 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:16:56.417039 3798194 cpu.go:282] Add success. I1122 06:16:56.441420 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:16:56.441986 3798194 net.go:642] Add success. I1122 06:16:56.443626 3798194 net.go:778] primary dev: ETH0 I1122 06:16:56.443643 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:16:56.443655 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:16:57.450913 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:16:57.451105 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:16:57.451148 3798194 disk_worker.go:723] disk space is not compliant W1122 06:16:57.451151 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:16:57.459832 3798194 disk_worker.go:509] system disk:vda1 I1122 06:16:57.459873 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:16:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:16:59.450010 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:16:59.450086 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:16:59.450127 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:16:59.471648 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:17:02.587287 3798194 disk_info.go:125] begin check local disk info of client I1122 06:17:02.591155 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:17:02.591164 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000372fc0 0xc000373000] E1122 06:17:06.413376 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:17:06.413399 3798194 memory.go:184] no items to output this cycle I1122 06:17:06.413545 3798194 cpu.go:275] no items to output this cycle E1122 06:17:16.413037 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:17:16.413059 3798194 memory.go:184] no items to output this cycle I1122 06:17:16.413233 3798194 cpu.go:275] no items to output this cycle E1122 06:17:26.411103 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:17:26.411155 3798194 memory.go:191] Add success. I1122 06:17:26.415617 3798194 cpu.go:282] Add success. I1122 06:17:26.441252 3798194 net.go:642] Add success. I1122 06:17:26.442858 3798194 net.go:778] primary dev: ETH0 I1122 06:17:26.442875 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:17:26.442888 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:17:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:17:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:17:29.450502 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:17:36.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:17:36.413260 3798194 memory.go:184] no items to output this cycle I1122 06:17:36.413402 3798194 cpu.go:275] no items to output this cycle E1122 06:17:46.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:17:46.412883 3798194 memory.go:184] no items to output this cycle I1122 06:17:46.413032 3798194 cpu.go:275] no items to output this cycle E1122 06:17:56.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:17:56.412273 3798194 memory.go:191] Add success. W1122 06:17:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:17:56.412311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:17:56.412314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:17:56.416938 3798194 cpu.go:282] Add success. I1122 06:17:56.441310 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:17:56.441782 3798194 net.go:642] Add success. I1122 06:17:56.443361 3798194 net.go:778] primary dev: ETH0 I1122 06:17:56.443377 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:17:56.443389 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:17:57.450610 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:17:57.450806 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:17:57.450817 3798194 disk_worker.go:723] disk space is not compliant W1122 06:17:57.450821 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:17:57.459150 3798194 disk_worker.go:509] system disk:vda1 I1122 06:17:57.459192 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:17:58.445700 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:17:59.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:17:59.450088 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:17:59.450123 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:17:59.471634 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:18:02.591267 3798194 disk_info.go:125] begin check local disk info of client I1122 06:18:02.595191 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:18:02.595200 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00044b140 0xc00044b180] E1122 06:18:06.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:18:06.411983 3798194 memory.go:184] no items to output this cycle I1122 06:18:06.412137 3798194 cpu.go:275] no items to output this cycle E1122 06:18:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:18:16.411976 3798194 memory.go:184] no items to output this cycle I1122 06:18:16.412138 3798194 cpu.go:275] no items to output this cycle I1122 06:18:17.642568 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:18:17.642576 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:18:26.411006 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:18:26.414139 3798194 memory.go:191] Add success. I1122 06:18:26.411184 3798194 cpu.go:282] Add success. I1122 06:18:26.441247 3798194 net.go:642] Add success. I1122 06:18:26.445454 3798194 net.go:778] primary dev: ETH0 I1122 06:18:26.445470 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:18:26.445483 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:18:29.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:18:29.450124 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:18:29.450152 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:18:36.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:18:36.413249 3798194 memory.go:184] no items to output this cycle I1122 06:18:36.413398 3798194 cpu.go:275] no items to output this cycle E1122 06:18:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:18:46.413261 3798194 memory.go:184] no items to output this cycle I1122 06:18:46.413403 3798194 cpu.go:275] no items to output this cycle E1122 06:18:56.412890 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:18:56.412923 3798194 memory.go:191] Add success. W1122 06:18:56.412950 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:18:56.412961 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:18:56.412965 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:18:56.417119 3798194 cpu.go:282] Add success. I1122 06:18:56.441432 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:18:56.442269 3798194 net.go:642] Add success. I1122 06:18:56.443817 3798194 net.go:778] primary dev: ETH0 I1122 06:18:56.443833 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:18:56.443847 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:18:57.450921 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:18:57.451151 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:18:57.451163 3798194 disk_worker.go:723] disk space is not compliant W1122 06:18:57.451166 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:18:57.459876 3798194 disk_worker.go:509] system disk:vda1 I1122 06:18:57.459918 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:18:58.446480 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:18:59.450587 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:18:59.450662 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:18:59.450689 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:18:59.471232 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:19:02.595285 3798194 disk_info.go:125] begin check local disk info of client I1122 06:19:02.599263 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:19:02.599272 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004b8bc0 0xc0004b8c00] E1122 06:19:06.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:19:06.413259 3798194 memory.go:184] no items to output this cycle I1122 06:19:06.413402 3798194 cpu.go:275] no items to output this cycle E1122 06:19:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:19:16.411980 3798194 memory.go:184] no items to output this cycle I1122 06:19:16.412143 3798194 cpu.go:275] no items to output this cycle E1122 06:19:26.412998 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:19:26.413033 3798194 memory.go:191] Add success. I1122 06:19:26.417307 3798194 cpu.go:282] Add success. I1122 06:19:26.441960 3798194 net.go:642] Add success. I1122 06:19:26.443620 3798194 net.go:778] primary dev: ETH0 I1122 06:19:26.443636 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:19:26.443649 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:19:29.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:19:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:19:29.450507 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:19:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:19:36.412256 3798194 memory.go:184] no items to output this cycle I1122 06:19:36.412400 3798194 cpu.go:275] no items to output this cycle E1122 06:19:46.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:19:46.413256 3798194 memory.go:184] no items to output this cycle I1122 06:19:46.413403 3798194 cpu.go:275] no items to output this cycle E1122 06:19:56.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:19:56.412277 3798194 memory.go:191] Add success. W1122 06:19:56.412310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:19:56.412322 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:19:56.412325 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:19:56.417101 3798194 cpu.go:282] Add success. I1122 06:19:56.441423 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:19:56.441910 3798194 net.go:642] Add success. I1122 06:19:56.443332 3798194 net.go:778] primary dev: ETH0 I1122 06:19:56.443347 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:19:56.443359 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:19:57.450725 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:19:57.450890 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:19:57.450944 3798194 disk_worker.go:723] disk space is not compliant W1122 06:19:57.450948 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:19:57.459760 3798194 disk_worker.go:509] system disk:vda1 I1122 06:19:57.459804 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:19:58.446281 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:19:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:19:59.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:19:59.450495 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:19:59.471006 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:20:02.599377 3798194 disk_info.go:125] begin check local disk info of client I1122 06:20:02.603275 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:20:02.603283 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003da8c0 0xc0003da900] E1122 06:20:06.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:20:06.412880 3798194 memory.go:184] no items to output this cycle I1122 06:20:06.413044 3798194 cpu.go:275] no items to output this cycle E1122 06:20:16.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:20:16.413259 3798194 memory.go:184] no items to output this cycle I1122 06:20:16.413413 3798194 cpu.go:275] no items to output this cycle E1122 06:20:26.413162 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:20:26.413195 3798194 memory.go:191] Add success. I1122 06:20:26.417368 3798194 cpu.go:282] Add success. I1122 06:20:26.441965 3798194 net.go:642] Add success. I1122 06:20:26.443464 3798194 net.go:778] primary dev: ETH0 I1122 06:20:26.443478 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:20:26.443490 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:20:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:20:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:20:29.450511 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:20:36.412047 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:20:36.412070 3798194 memory.go:184] no items to output this cycle I1122 06:20:36.412248 3798194 cpu.go:275] no items to output this cycle E1122 06:20:46.413524 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:20:46.413555 3798194 memory.go:184] no items to output this cycle I1122 06:20:46.413710 3798194 cpu.go:275] no items to output this cycle E1122 06:20:56.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:20:56.412893 3798194 memory.go:191] Add success. W1122 06:20:56.412923 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:20:56.412945 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:20:56.412949 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:20:56.417504 3798194 cpu.go:282] Add success. I1122 06:20:56.445853 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:20:56.446394 3798194 net.go:642] Add success. I1122 06:20:56.447958 3798194 net.go:778] primary dev: ETH0 I1122 06:20:56.447972 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:20:56.447987 3798194 net.go:659] Add eni metrics success, count: 7 E1122 06:20:57.450636 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 06:20:57.450650 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:20:57.450848 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:20:57.450857 3798194 disk_worker.go:723] disk space is not compliant W1122 06:20:57.450860 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:20:57.463889 3798194 disk_worker.go:509] system disk:vda1 I1122 06:20:57.463935 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 06:20:58.448159 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 06:20:58.448174 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 06:20:59.448638 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 06:20:59.449720 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:20:59.449790 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:20:59.449810 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:20:59.471348 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:21:02.603388 3798194 disk_info.go:125] begin check local disk info of client I1122 06:21:02.607273 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:21:02.607284 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036e4c0 0xc00036e500] E1122 06:21:06.412349 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:21:06.412373 3798194 memory.go:184] no items to output this cycle I1122 06:21:06.412514 3798194 cpu.go:275] no items to output this cycle E1122 06:21:16.412333 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:21:16.412354 3798194 memory.go:184] no items to output this cycle I1122 06:21:16.412506 3798194 cpu.go:275] no items to output this cycle I1122 06:21:17.642757 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:21:17.642765 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:21:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:21:26.413268 3798194 memory.go:191] Add success. I1122 06:21:26.417395 3798194 cpu.go:282] Add success. I1122 06:21:26.441915 3798194 net.go:642] Add success. I1122 06:21:26.443314 3798194 net.go:778] primary dev: ETH0 I1122 06:21:26.443329 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:21:26.443341 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:21:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:21:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:21:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:21:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:21:36.411982 3798194 memory.go:184] no items to output this cycle I1122 06:21:36.412341 3798194 cpu.go:275] no items to output this cycle E1122 06:21:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:21:46.411989 3798194 memory.go:184] no items to output this cycle I1122 06:21:46.412152 3798194 cpu.go:275] no items to output this cycle E1122 06:21:56.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:21:56.411958 3798194 memory.go:191] Add success. W1122 06:21:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:21:56.412002 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:21:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:21:56.412393 3798194 cpu.go:282] Add success. I1122 06:21:56.441847 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:21:56.442359 3798194 net.go:642] Add success. I1122 06:21:56.446469 3798194 net.go:778] primary dev: ETH0 I1122 06:21:56.446484 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:21:56.446496 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:21:57.451196 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:21:57.451372 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:21:57.451429 3798194 disk_worker.go:723] disk space is not compliant W1122 06:21:57.451432 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:21:57.459841 3798194 disk_worker.go:509] system disk:vda1 I1122 06:21:57.459881 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:21:58.446032 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:21:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:21:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:21:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:21:59.471671 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:22:02.607389 3798194 disk_info.go:125] begin check local disk info of client I1122 06:22:02.611216 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:22:02.611224 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00045f300 0xc00045f340] E1122 06:22:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:22:06.413255 3798194 memory.go:184] no items to output this cycle I1122 06:22:06.413398 3798194 cpu.go:275] no items to output this cycle E1122 06:22:16.413250 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:22:16.413272 3798194 memory.go:184] no items to output this cycle I1122 06:22:16.413421 3798194 cpu.go:275] no items to output this cycle E1122 06:22:26.412288 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:22:26.412335 3798194 memory.go:191] Add success. I1122 06:22:26.417440 3798194 cpu.go:282] Add success. I1122 06:22:26.441599 3798194 net.go:642] Add success. I1122 06:22:26.443312 3798194 net.go:778] primary dev: ETH0 I1122 06:22:26.443334 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:22:26.443357 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:22:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:22:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:22:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:22:36.412913 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:22:36.412936 3798194 memory.go:184] no items to output this cycle I1122 06:22:36.413089 3798194 cpu.go:275] no items to output this cycle E1122 06:22:46.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:22:46.411986 3798194 memory.go:184] no items to output this cycle I1122 06:22:46.412145 3798194 cpu.go:275] no items to output this cycle E1122 06:22:56.413052 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:22:56.413086 3798194 memory.go:191] Add success. W1122 06:22:56.413141 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:22:56.413155 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:22:56.413159 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:22:56.417430 3798194 cpu.go:282] Add success. I1122 06:22:56.441806 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:22:56.442561 3798194 net.go:642] Add success. I1122 06:22:56.444129 3798194 net.go:778] primary dev: ETH0 I1122 06:22:56.444145 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:22:56.444164 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:22:57.451305 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:22:57.451472 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:22:57.451529 3798194 disk_worker.go:723] disk space is not compliant W1122 06:22:57.451532 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:22:57.459875 3798194 disk_worker.go:509] system disk:vda1 I1122 06:22:57.459915 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:22:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:22:59.450650 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:22:59.450731 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:22:59.450755 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:22:59.471185 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:23:02.611325 3798194 disk_info.go:125] begin check local disk info of client I1122 06:23:02.615217 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:23:02.615226 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000070ec0 0xc000070f00] E1122 06:23:06.412048 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:23:06.412071 3798194 memory.go:184] no items to output this cycle I1122 06:23:06.412244 3798194 cpu.go:275] no items to output this cycle E1122 06:23:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:23:16.413256 3798194 memory.go:184] no items to output this cycle I1122 06:23:16.413398 3798194 cpu.go:275] no items to output this cycle E1122 06:23:26.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:23:26.413273 3798194 memory.go:191] Add success. I1122 06:23:26.417585 3798194 cpu.go:282] Add success. I1122 06:23:26.442106 3798194 net.go:642] Add success. I1122 06:23:26.443623 3798194 net.go:778] primary dev: ETH0 I1122 06:23:26.443636 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:23:26.443649 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:23:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:23:29.450088 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:23:29.450122 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:23:36.413566 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:23:36.413591 3798194 memory.go:184] no items to output this cycle I1122 06:23:36.413737 3798194 cpu.go:275] no items to output this cycle E1122 06:23:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:23:46.413259 3798194 memory.go:184] no items to output this cycle I1122 06:23:46.413410 3798194 cpu.go:275] no items to output this cycle E1122 06:23:56.412846 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:23:56.412888 3798194 memory.go:191] Add success. W1122 06:23:56.412915 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:23:56.412927 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:23:56.412930 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:23:56.417638 3798194 cpu.go:282] Add success. I1122 06:23:56.440951 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:23:56.441552 3798194 net.go:642] Add success. I1122 06:23:56.443169 3798194 net.go:778] primary dev: ETH0 I1122 06:23:56.443185 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:23:56.443197 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:23:57.450688 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:23:57.450879 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:23:57.450890 3798194 disk_worker.go:723] disk space is not compliant W1122 06:23:57.450893 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:23:57.459282 3798194 disk_worker.go:509] system disk:vda1 I1122 06:23:57.459326 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:23:58.445999 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:23:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:23:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:23:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:23:59.471672 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:24:02.615320 3798194 disk_info.go:125] begin check local disk info of client I1122 06:24:02.619189 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:24:02.619198 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00048a0c0 0xc00048a100] E1122 06:24:06.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:24:06.412873 3798194 memory.go:184] no items to output this cycle I1122 06:24:06.413012 3798194 cpu.go:275] no items to output this cycle E1122 06:24:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:24:16.411982 3798194 memory.go:184] no items to output this cycle I1122 06:24:16.412147 3798194 cpu.go:275] no items to output this cycle I1122 06:24:17.650207 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:24:17.650216 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:24:26.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:24:26.413266 3798194 memory.go:191] Add success. I1122 06:24:26.417578 3798194 cpu.go:282] Add success. I1122 06:24:26.442133 3798194 net.go:642] Add success. I1122 06:24:26.443905 3798194 net.go:778] primary dev: ETH0 I1122 06:24:26.443920 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:24:26.443933 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:24:29.449955 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:24:29.450034 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:24:29.450066 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:24:36.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:24:36.411987 3798194 memory.go:184] no items to output this cycle I1122 06:24:36.412136 3798194 cpu.go:275] no items to output this cycle E1122 06:24:46.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:24:46.412873 3798194 memory.go:184] no items to output this cycle I1122 06:24:46.413016 3798194 cpu.go:275] no items to output this cycle E1122 06:24:56.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:24:56.413272 3798194 memory.go:191] Add success. W1122 06:24:56.413300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:24:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:24:56.413315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:24:56.417508 3798194 cpu.go:282] Add success. I1122 06:24:56.441895 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:24:56.442417 3798194 net.go:642] Add success. I1122 06:24:56.444284 3798194 net.go:778] primary dev: ETH0 I1122 06:24:56.444299 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:24:56.444312 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:24:57.450989 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:24:57.451183 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:24:57.451236 3798194 disk_worker.go:723] disk space is not compliant W1122 06:24:57.451239 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:24:57.460013 3798194 disk_worker.go:509] system disk:vda1 I1122 06:24:57.460053 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:24:58.445952 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:24:59.450052 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:24:59.450145 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:24:59.450171 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:24:59.471694 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:25:02.619307 3798194 disk_info.go:125] begin check local disk info of client I1122 06:25:02.623083 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:25:02.623092 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000372380 0xc0003723c0] E1122 06:25:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:25:06.412254 3798194 memory.go:184] no items to output this cycle I1122 06:25:06.412396 3798194 cpu.go:275] no items to output this cycle E1122 06:25:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:25:16.413258 3798194 memory.go:184] no items to output this cycle I1122 06:25:16.413401 3798194 cpu.go:275] no items to output this cycle E1122 06:25:26.411188 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:25:26.411221 3798194 memory.go:191] Add success. I1122 06:25:26.415391 3798194 cpu.go:282] Add success. I1122 06:25:26.441942 3798194 net.go:642] Add success. I1122 06:25:26.443514 3798194 net.go:778] primary dev: ETH0 I1122 06:25:26.443529 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:25:26.443540 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:25:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:25:29.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:25:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:25:36.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:25:36.411972 3798194 memory.go:184] no items to output this cycle I1122 06:25:36.412123 3798194 cpu.go:275] no items to output this cycle E1122 06:25:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:25:46.411974 3798194 memory.go:184] no items to output this cycle I1122 06:25:46.412328 3798194 cpu.go:275] no items to output this cycle E1122 06:25:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:25:56.411944 3798194 memory.go:191] Add success. W1122 06:25:56.411971 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:25:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:25:56.411986 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:25:56.412360 3798194 cpu.go:282] Add success. I1122 06:25:56.441588 3798194 event_worker.go:152] Polling the log file for events... I1122 06:25:56.444699 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:25:56.445198 3798194 net.go:642] Add success. I1122 06:25:56.448964 3798194 net.go:778] primary dev: ETH0 I1122 06:25:56.448978 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:25:56.448991 3798194 net.go:659] Add eni metrics success, count: 7 E1122 06:25:57.450666 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 06:25:57.450680 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:25:57.450833 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:25:57.450883 3798194 disk_worker.go:723] disk space is not compliant W1122 06:25:57.450886 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:25:57.459352 3798194 disk_worker.go:509] system disk:vda1 I1122 06:25:57.459393 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 06:25:58.448086 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 06:25:58.448100 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 06:25:59.448604 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 06:25:59.449678 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:25:59.449731 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:25:59.449749 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:25:59.471268 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:26:02.623194 3798194 disk_info.go:125] begin check local disk info of client I1122 06:26:02.626978 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:26:02.626987 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041e6c0 0xc00041e700] E1122 06:26:06.412754 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:26:06.412775 3798194 memory.go:184] no items to output this cycle I1122 06:26:06.412930 3798194 cpu.go:275] no items to output this cycle E1122 06:26:16.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:26:16.411993 3798194 memory.go:184] no items to output this cycle I1122 06:26:16.412155 3798194 cpu.go:275] no items to output this cycle E1122 06:26:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:26:26.413276 3798194 memory.go:191] Add success. I1122 06:26:26.417163 3798194 cpu.go:282] Add success. I1122 06:26:26.441673 3798194 net.go:642] Add success. I1122 06:26:26.443233 3798194 net.go:778] primary dev: ETH0 I1122 06:26:26.443248 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:26:26.443261 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:26:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:26:29.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:26:29.450512 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:26:36.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:26:36.413271 3798194 memory.go:184] no items to output this cycle I1122 06:26:36.413412 3798194 cpu.go:275] no items to output this cycle E1122 06:26:46.411301 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:26:46.411326 3798194 memory.go:184] no items to output this cycle I1122 06:26:46.411474 3798194 cpu.go:275] no items to output this cycle E1122 06:26:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:26:56.411958 3798194 memory.go:191] Add success. W1122 06:26:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:26:56.412001 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:26:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:26:56.412354 3798194 cpu.go:282] Add success. I1122 06:26:56.441821 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:26:56.442332 3798194 net.go:642] Add success. I1122 06:26:56.446449 3798194 net.go:778] primary dev: ETH0 I1122 06:26:56.446465 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:26:56.446478 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:26:57.450911 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:26:57.451143 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:26:57.451154 3798194 disk_worker.go:723] disk space is not compliant W1122 06:26:57.451157 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:26:57.466757 3798194 disk_worker.go:509] system disk:vda1 I1122 06:26:57.466802 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:26:58.445991 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:26:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:26:59.450130 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:26:59.450157 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:26:59.471718 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:27:02.627092 3798194 disk_info.go:125] begin check local disk info of client I1122 06:27:02.630906 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:27:02.630915 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000465340 0xc000465540] E1122 06:27:06.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:27:06.413254 3798194 memory.go:184] no items to output this cycle I1122 06:27:06.413396 3798194 cpu.go:275] no items to output this cycle E1122 06:27:16.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:27:16.413259 3798194 memory.go:184] no items to output this cycle I1122 06:27:16.413415 3798194 cpu.go:275] no items to output this cycle I1122 06:27:17.650738 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:27:17.650746 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:27:26.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:27:26.412278 3798194 memory.go:191] Add success. I1122 06:27:26.416639 3798194 cpu.go:282] Add success. I1122 06:27:26.442170 3798194 net.go:642] Add success. I1122 06:27:26.443556 3798194 net.go:778] primary dev: ETH0 I1122 06:27:26.443571 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:27:26.443587 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:27:29.450198 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:27:29.450280 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:27:29.450307 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:27:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:27:36.411980 3798194 memory.go:184] no items to output this cycle I1122 06:27:36.412153 3798194 cpu.go:275] no items to output this cycle E1122 06:27:46.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:27:46.412268 3798194 memory.go:184] no items to output this cycle I1122 06:27:46.412418 3798194 cpu.go:275] no items to output this cycle E1122 06:27:56.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:27:56.412277 3798194 memory.go:191] Add success. W1122 06:27:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:27:56.412322 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:27:56.412325 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:27:56.417079 3798194 cpu.go:282] Add success. I1122 06:27:56.441601 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:27:56.442177 3798194 net.go:642] Add success. I1122 06:27:56.443668 3798194 net.go:778] primary dev: ETH0 I1122 06:27:56.443682 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:27:56.443694 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:27:57.452007 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:27:57.452195 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:27:57.452252 3798194 disk_worker.go:723] disk space is not compliant W1122 06:27:57.452255 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:27:57.461020 3798194 disk_worker.go:509] system disk:vda1 I1122 06:27:57.461064 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:27:58.446663 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:27:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:27:59.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:27:59.450508 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:27:59.471039 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:28:02.631135 3798194 disk_info.go:125] begin check local disk info of client I1122 06:28:02.634972 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:28:02.634982 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b8b80 0xc0003b8bc0] E1122 06:28:06.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:28:06.412250 3798194 memory.go:184] no items to output this cycle I1122 06:28:06.412404 3798194 cpu.go:275] no items to output this cycle E1122 06:28:16.413002 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:28:16.413024 3798194 memory.go:184] no items to output this cycle I1122 06:28:16.413191 3798194 cpu.go:275] no items to output this cycle E1122 06:28:26.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:28:26.412278 3798194 memory.go:191] Add success. I1122 06:28:26.416687 3798194 cpu.go:282] Add success. I1122 06:28:26.442283 3798194 net.go:642] Add success. I1122 06:28:26.443720 3798194 net.go:778] primary dev: ETH0 I1122 06:28:26.443736 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:28:26.443749 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:28:29.450029 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:28:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:28:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:28:36.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:28:36.411970 3798194 memory.go:184] no items to output this cycle I1122 06:28:36.412125 3798194 cpu.go:275] no items to output this cycle E1122 06:28:46.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:28:46.412250 3798194 memory.go:184] no items to output this cycle I1122 06:28:46.412398 3798194 cpu.go:275] no items to output this cycle E1122 06:28:56.412035 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:28:56.412068 3798194 memory.go:191] Add success. W1122 06:28:56.412096 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:28:56.412126 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:28:56.412132 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:28:56.416493 3798194 cpu.go:282] Add success. I1122 06:28:56.441781 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:28:56.442427 3798194 net.go:642] Add success. I1122 06:28:56.443880 3798194 net.go:778] primary dev: ETH0 I1122 06:28:56.443894 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:28:56.443908 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:28:57.450644 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:28:57.450803 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:28:57.450855 3798194 disk_worker.go:723] disk space is not compliant W1122 06:28:57.450858 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:28:57.459272 3798194 disk_worker.go:509] system disk:vda1 I1122 06:28:57.459313 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:28:58.446027 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:28:59.450408 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:28:59.450508 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:28:59.450536 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:28:59.471061 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:29:02.635092 3798194 disk_info.go:125] begin check local disk info of client I1122 06:29:02.638981 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:29:02.638989 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000315400 0xc000315440] E1122 06:29:06.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:29:06.412266 3798194 memory.go:184] no items to output this cycle I1122 06:29:06.412409 3798194 cpu.go:275] no items to output this cycle E1122 06:29:16.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:29:16.412254 3798194 memory.go:184] no items to output this cycle I1122 06:29:16.412399 3798194 cpu.go:275] no items to output this cycle E1122 06:29:26.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:29:26.412269 3798194 memory.go:191] Add success. I1122 06:29:26.416293 3798194 cpu.go:282] Add success. I1122 06:29:26.441964 3798194 net.go:642] Add success. I1122 06:29:26.443444 3798194 net.go:778] primary dev: ETH0 I1122 06:29:26.443459 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:29:26.443474 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:29:29.450194 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:29:29.450279 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:29:29.450303 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:29:36.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:29:36.412263 3798194 memory.go:184] no items to output this cycle I1122 06:29:36.412407 3798194 cpu.go:275] no items to output this cycle E1122 06:29:46.411975 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:29:46.412009 3798194 memory.go:184] no items to output this cycle I1122 06:29:46.412392 3798194 cpu.go:275] no items to output this cycle E1122 06:29:56.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:29:56.413263 3798194 memory.go:191] Add success. W1122 06:29:56.413299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:29:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:29:56.413316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:29:56.418135 3798194 cpu.go:282] Add success. I1122 06:29:56.441445 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:29:56.442021 3798194 net.go:642] Add success. I1122 06:29:56.443563 3798194 net.go:778] primary dev: ETH0 I1122 06:29:56.443578 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:29:56.443593 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:29:57.445489 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:29:57.445789 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:29:57.445801 3798194 disk_worker.go:723] disk space is not compliant W1122 06:29:57.445805 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:29:57.448790 3798194 disk_worker.go:509] system disk:vda1 I1122 06:29:57.448831 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:29:58.446387 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:29:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:29:59.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:29:59.450511 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:29:59.471032 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:30:02.639103 3798194 disk_info.go:125] begin check local disk info of client I1122 06:30:02.642953 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:30:02.642963 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002f2d80 0xc0002f2dc0] E1122 06:30:06.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:30:06.412262 3798194 memory.go:184] no items to output this cycle I1122 06:30:06.412408 3798194 cpu.go:275] no items to output this cycle E1122 06:30:16.412057 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:30:16.412080 3798194 memory.go:184] no items to output this cycle I1122 06:30:16.412248 3798194 cpu.go:275] no items to output this cycle I1122 06:30:17.651198 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:30:17.651207 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:30:26.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:30:26.412271 3798194 memory.go:191] Add success. I1122 06:30:26.416372 3798194 cpu.go:282] Add success. I1122 06:30:26.441958 3798194 net.go:642] Add success. I1122 06:30:26.443438 3798194 net.go:778] primary dev: ETH0 I1122 06:30:26.443452 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:30:26.443464 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:30:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:30:29.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:30:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:30:36.412462 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:30:36.412485 3798194 memory.go:184] no items to output this cycle I1122 06:30:36.412628 3798194 cpu.go:275] no items to output this cycle E1122 06:30:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:30:46.413259 3798194 memory.go:184] no items to output this cycle I1122 06:30:46.413410 3798194 cpu.go:275] no items to output this cycle E1122 06:30:56.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:30:56.412269 3798194 memory.go:191] Add success. W1122 06:30:56.412298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:30:56.412310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:30:56.412313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:30:56.416620 3798194 cpu.go:282] Add success. I1122 06:30:56.445415 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:30:56.446002 3798194 net.go:642] Add success. I1122 06:30:56.447488 3798194 net.go:778] primary dev: ETH0 I1122 06:30:56.447504 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:30:56.447517 3798194 net.go:659] Add eni metrics success, count: 7 E1122 06:30:57.451099 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 06:30:57.451131 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:30:57.451407 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:30:57.451468 3798194 disk_worker.go:723] disk space is not compliant W1122 06:30:57.451471 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:30:57.461068 3798194 disk_worker.go:509] system disk:vda1 I1122 06:30:57.461120 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 06:30:58.448078 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 06:30:58.448093 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 06:30:59.448625 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 06:30:59.449700 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:30:59.449751 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:30:59.449772 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:30:59.471315 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:31:02.643195 3798194 disk_info.go:125] begin check local disk info of client I1122 06:31:02.647007 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:31:02.647015 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041d600 0xc00041d640] E1122 06:31:06.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:31:06.412254 3798194 memory.go:184] no items to output this cycle I1122 06:31:06.412396 3798194 cpu.go:275] no items to output this cycle E1122 06:31:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:31:16.412260 3798194 memory.go:184] no items to output this cycle I1122 06:31:16.412412 3798194 cpu.go:275] no items to output this cycle E1122 06:31:26.412288 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:31:26.412344 3798194 memory.go:191] Add success. I1122 06:31:26.429084 3798194 cpu.go:282] Add success. I1122 06:31:26.441295 3798194 net.go:642] Add success. I1122 06:31:26.442856 3798194 net.go:778] primary dev: ETH0 I1122 06:31:26.442871 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:31:26.442883 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:31:29.450035 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:31:29.450154 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:31:29.450180 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:31:36.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:31:36.411985 3798194 memory.go:184] no items to output this cycle I1122 06:31:36.412143 3798194 cpu.go:275] no items to output this cycle E1122 06:31:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:31:46.411974 3798194 memory.go:184] no items to output this cycle I1122 06:31:46.412142 3798194 cpu.go:275] no items to output this cycle E1122 06:31:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:31:56.413262 3798194 memory.go:191] Add success. W1122 06:31:56.413291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:31:56.413302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:31:56.413305 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:31:56.417786 3798194 cpu.go:282] Add success. I1122 06:31:56.441162 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:31:56.441733 3798194 net.go:642] Add success. I1122 06:31:56.443165 3798194 net.go:778] primary dev: ETH0 I1122 06:31:56.443179 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:31:56.443191 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:31:57.450967 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:31:57.451205 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:31:57.451218 3798194 disk_worker.go:723] disk space is not compliant W1122 06:31:57.451221 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:31:57.460013 3798194 disk_worker.go:509] system disk:vda1 I1122 06:31:57.460056 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:31:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:31:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:31:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:31:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:31:59.471681 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:32:02.647120 3798194 disk_info.go:125] begin check local disk info of client I1122 06:32:02.650957 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:32:02.650965 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f2d00 0xc0003f2d40] E1122 06:32:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:32:06.411985 3798194 memory.go:184] no items to output this cycle I1122 06:32:06.412140 3798194 cpu.go:275] no items to output this cycle E1122 06:32:16.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:32:16.413268 3798194 memory.go:184] no items to output this cycle I1122 06:32:16.413423 3798194 cpu.go:275] no items to output this cycle E1122 06:32:26.417304 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:32:26.417359 3798194 memory.go:191] Add success. I1122 06:32:26.422064 3798194 cpu.go:282] Add success. I1122 06:32:26.442490 3798194 net.go:642] Add success. I1122 06:32:26.444230 3798194 net.go:778] primary dev: ETH0 I1122 06:32:26.444255 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:32:26.444275 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:32:29.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:32:29.450489 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:32:29.450519 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:32:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:32:36.411977 3798194 memory.go:184] no items to output this cycle I1122 06:32:36.412131 3798194 cpu.go:275] no items to output this cycle E1122 06:32:46.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:32:46.412872 3798194 memory.go:184] no items to output this cycle I1122 06:32:46.413020 3798194 cpu.go:275] no items to output this cycle E1122 06:32:56.413255 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:32:56.413289 3798194 memory.go:191] Add success. W1122 06:32:56.413320 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:32:56.413332 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:32:56.413336 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:32:56.417642 3798194 cpu.go:282] Add success. I1122 06:32:56.441365 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:32:56.441920 3798194 net.go:642] Add success. I1122 06:32:56.443320 3798194 net.go:778] primary dev: ETH0 I1122 06:32:56.443336 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:32:56.443347 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:32:57.446285 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:32:57.446505 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:32:57.446584 3798194 disk_worker.go:723] disk space is not compliant W1122 06:32:57.446587 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:32:57.451664 3798194 disk_worker.go:509] system disk:vda1 I1122 06:32:57.451704 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:32:58.446377 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:32:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:32:59.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:32:59.450131 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:32:59.471654 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:33:02.651066 3798194 disk_info.go:125] begin check local disk info of client I1122 06:33:02.654909 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:33:02.654917 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00027d440 0xc00027d480] E1122 06:33:06.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:33:06.413260 3798194 memory.go:184] no items to output this cycle I1122 06:33:06.413410 3798194 cpu.go:275] no items to output this cycle E1122 06:33:16.412835 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:33:16.412857 3798194 memory.go:184] no items to output this cycle I1122 06:33:16.413002 3798194 cpu.go:275] no items to output this cycle I1122 06:33:17.651366 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:33:17.651374 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:33:26.412901 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:33:26.412932 3798194 memory.go:191] Add success. I1122 06:33:26.417324 3798194 cpu.go:282] Add success. I1122 06:33:26.441964 3798194 net.go:642] Add success. I1122 06:33:26.443460 3798194 net.go:778] primary dev: ETH0 I1122 06:33:26.443475 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:33:26.443493 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:33:29.450232 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:33:29.450315 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:33:29.450340 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:33:36.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:33:36.412264 3798194 memory.go:184] no items to output this cycle I1122 06:33:36.412407 3798194 cpu.go:275] no items to output this cycle E1122 06:33:46.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:33:46.412902 3798194 memory.go:184] no items to output this cycle I1122 06:33:46.413072 3798194 cpu.go:275] no items to output this cycle E1122 06:33:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:33:56.411961 3798194 memory.go:191] Add success. W1122 06:33:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:33:56.412001 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:33:56.412005 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:33:56.412377 3798194 cpu.go:282] Add success. I1122 06:33:56.441945 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:33:56.442556 3798194 net.go:642] Add success. I1122 06:33:56.446661 3798194 net.go:778] primary dev: ETH0 I1122 06:33:56.446676 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:33:56.446688 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:33:57.450924 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:33:57.451151 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:33:57.451162 3798194 disk_worker.go:723] disk space is not compliant W1122 06:33:57.451165 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:33:57.459899 3798194 disk_worker.go:509] system disk:vda1 I1122 06:33:57.459940 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:33:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:33:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:33:59.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:33:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:33:59.471722 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:34:02.655143 3798194 disk_info.go:125] begin check local disk info of client I1122 06:34:02.658989 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:34:02.659000 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0dc0 0xc0002e0e00] E1122 06:34:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:34:06.411976 3798194 memory.go:184] no items to output this cycle I1122 06:34:06.412133 3798194 cpu.go:275] no items to output this cycle E1122 06:34:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:34:16.411978 3798194 memory.go:184] no items to output this cycle I1122 06:34:16.412137 3798194 cpu.go:275] no items to output this cycle E1122 06:34:26.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:34:26.412883 3798194 memory.go:191] Add success. I1122 06:34:26.417356 3798194 cpu.go:282] Add success. I1122 06:34:26.442195 3798194 net.go:642] Add success. I1122 06:34:26.443602 3798194 net.go:778] primary dev: ETH0 I1122 06:34:26.443624 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:34:26.443638 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:34:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:34:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:34:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:34:36.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:34:36.413254 3798194 memory.go:184] no items to output this cycle I1122 06:34:36.413402 3798194 cpu.go:275] no items to output this cycle E1122 06:34:46.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:34:46.412250 3798194 memory.go:184] no items to output this cycle I1122 06:34:46.412393 3798194 cpu.go:275] no items to output this cycle E1122 06:34:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:34:56.411956 3798194 memory.go:191] Add success. W1122 06:34:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:34:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:34:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:34:56.412350 3798194 cpu.go:282] Add success. I1122 06:34:56.440942 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:34:56.441538 3798194 net.go:642] Add success. I1122 06:34:56.447342 3798194 net.go:778] primary dev: ETH0 I1122 06:34:56.447357 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:34:56.447371 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:34:57.445779 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:34:57.446039 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:34:57.446050 3798194 disk_worker.go:723] disk space is not compliant W1122 06:34:57.446054 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:34:57.449567 3798194 disk_worker.go:509] system disk:vda1 I1122 06:34:57.449611 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:34:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:34:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:34:59.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:34:59.450136 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:34:59.471642 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:35:02.659144 3798194 disk_info.go:125] begin check local disk info of client I1122 06:35:02.663007 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:35:02.663015 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046a3c0 0xc00046a400] E1122 06:35:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:35:06.411981 3798194 memory.go:184] no items to output this cycle I1122 06:35:06.412136 3798194 cpu.go:275] no items to output this cycle E1122 06:35:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:35:16.411980 3798194 memory.go:184] no items to output this cycle I1122 06:35:16.412141 3798194 cpu.go:275] no items to output this cycle E1122 06:35:26.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:35:26.411948 3798194 memory.go:191] Add success. I1122 06:35:26.412324 3798194 cpu.go:282] Add success. I1122 06:35:26.442084 3798194 net.go:642] Add success. I1122 06:35:26.446033 3798194 net.go:778] primary dev: ETH0 I1122 06:35:26.446057 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:35:26.446071 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:35:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:35:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:35:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:35:36.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:35:36.411971 3798194 memory.go:184] no items to output this cycle I1122 06:35:36.412118 3798194 cpu.go:275] no items to output this cycle E1122 06:35:46.412737 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:35:46.412760 3798194 memory.go:184] no items to output this cycle I1122 06:35:46.412915 3798194 cpu.go:275] no items to output this cycle E1122 06:35:56.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:35:56.411947 3798194 memory.go:191] Add success. W1122 06:35:56.411974 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:35:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:35:56.411989 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:35:56.412347 3798194 cpu.go:282] Add success. I1122 06:35:56.441644 3798194 event_worker.go:152] Polling the log file for events... I1122 06:35:56.449736 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:35:56.450291 3798194 net.go:642] Add success. I1122 06:35:56.454574 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795298756, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 06:35:56.454684 3798194 net.go:778] primary dev: ETH0 I1122 06:35:56.454697 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:35:56.454708 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:35:56.465742 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 06:35:57.446396 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 06:35:57.446410 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:35:57.446605 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:35:57.446667 3798194 disk_worker.go:723] disk space is not compliant W1122 06:35:57.446670 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:35:57.449468 3798194 disk_worker.go:509] system disk:vda1 I1122 06:35:57.449507 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 06:35:58.448173 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 06:35:58.448187 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 06:35:59.448974 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 06:35:59.450047 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:35:59.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:35:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:35:59.471662 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:36:02.663106 3798194 disk_info.go:125] begin check local disk info of client I1122 06:36:02.666906 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:36:02.666915 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000395c80 0xc000395cc0] E1122 06:36:06.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:36:06.413249 3798194 memory.go:184] no items to output this cycle I1122 06:36:06.413390 3798194 cpu.go:275] no items to output this cycle E1122 06:36:16.412028 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:36:16.412049 3798194 memory.go:184] no items to output this cycle I1122 06:36:16.412218 3798194 cpu.go:275] no items to output this cycle I1122 06:36:17.651534 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:36:17.651542 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:36:26.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:36:26.412889 3798194 memory.go:191] Add success. I1122 06:36:26.417076 3798194 cpu.go:282] Add success. I1122 06:36:26.442016 3798194 net.go:642] Add success. I1122 06:36:26.443463 3798194 net.go:778] primary dev: ETH0 I1122 06:36:26.443492 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:36:26.443506 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:36:29.449675 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:36:29.449748 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:36:29.449782 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:36:36.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:36:36.411987 3798194 memory.go:184] no items to output this cycle I1122 06:36:36.412142 3798194 cpu.go:275] no items to output this cycle E1122 06:36:46.412245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:36:46.412266 3798194 memory.go:184] no items to output this cycle I1122 06:36:46.412440 3798194 cpu.go:275] no items to output this cycle E1122 06:36:56.412252 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:36:56.412286 3798194 memory.go:191] Add success. W1122 06:36:56.412317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:36:56.412328 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:36:56.412331 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:36:56.416503 3798194 cpu.go:282] Add success. I1122 06:36:56.441948 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:36:56.442563 3798194 net.go:642] Add success. I1122 06:36:56.444021 3798194 net.go:778] primary dev: ETH0 I1122 06:36:56.444035 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:36:56.444048 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:36:57.450663 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:36:57.450830 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:36:57.450886 3798194 disk_worker.go:723] disk space is not compliant W1122 06:36:57.450889 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:36:57.459361 3798194 disk_worker.go:509] system disk:vda1 I1122 06:36:57.459403 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:36:58.446009 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:36:59.450122 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:36:59.450199 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:36:59.450225 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:36:59.470720 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:37:02.667017 3798194 disk_info.go:125] begin check local disk info of client I1122 06:37:02.670883 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:37:02.670893 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00043e180 0xc00043e1c0] E1122 06:37:06.412390 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:37:06.412416 3798194 memory.go:184] no items to output this cycle I1122 06:37:06.412566 3798194 cpu.go:275] no items to output this cycle E1122 06:37:16.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:37:16.412266 3798194 memory.go:184] no items to output this cycle I1122 06:37:16.412413 3798194 cpu.go:275] no items to output this cycle E1122 06:37:26.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:37:26.411952 3798194 memory.go:191] Add success. I1122 06:37:26.412317 3798194 cpu.go:282] Add success. I1122 06:37:26.441943 3798194 net.go:642] Add success. I1122 06:37:26.446327 3798194 net.go:778] primary dev: ETH0 I1122 06:37:26.446343 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:37:26.446357 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:37:29.450259 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:37:29.450340 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:37:29.450366 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:37:36.412842 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:37:36.412864 3798194 memory.go:184] no items to output this cycle I1122 06:37:36.413007 3798194 cpu.go:275] no items to output this cycle E1122 06:37:46.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:37:46.413261 3798194 memory.go:184] no items to output this cycle I1122 06:37:46.413405 3798194 cpu.go:275] no items to output this cycle E1122 06:37:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:37:56.411955 3798194 memory.go:191] Add success. W1122 06:37:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:37:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:37:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:37:56.412373 3798194 cpu.go:282] Add success. I1122 06:37:56.442095 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:37:56.443080 3798194 net.go:642] Add success. I1122 06:37:56.447044 3798194 net.go:778] primary dev: ETH0 I1122 06:37:56.447060 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:37:56.447074 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:37:57.452438 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:37:57.452602 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:37:57.452660 3798194 disk_worker.go:723] disk space is not compliant W1122 06:37:57.452664 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:37:57.468537 3798194 disk_worker.go:509] system disk:vda1 I1122 06:37:57.468583 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:37:58.445956 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:37:59.450053 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:37:59.450140 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:37:59.450166 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:37:59.471698 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:38:02.672005 3798194 disk_info.go:125] begin check local disk info of client I1122 06:38:02.675825 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:38:02.675834 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004fc300 0xc0004fc340] E1122 06:38:06.412775 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:38:06.412797 3798194 memory.go:184] no items to output this cycle I1122 06:38:06.412952 3798194 cpu.go:275] no items to output this cycle E1122 06:38:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:38:16.413252 3798194 memory.go:184] no items to output this cycle I1122 06:38:16.413405 3798194 cpu.go:275] no items to output this cycle E1122 06:38:26.411909 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:38:26.411942 3798194 memory.go:191] Add success. I1122 06:38:26.412300 3798194 cpu.go:282] Add success. I1122 06:38:26.441935 3798194 net.go:642] Add success. I1122 06:38:26.446275 3798194 net.go:778] primary dev: ETH0 I1122 06:38:26.446289 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:38:26.446302 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:38:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:38:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:38:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:38:36.412944 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:38:36.412966 3798194 memory.go:184] no items to output this cycle I1122 06:38:36.413123 3798194 cpu.go:275] no items to output this cycle E1122 06:38:46.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:38:46.412267 3798194 memory.go:184] no items to output this cycle I1122 06:38:46.412410 3798194 cpu.go:275] no items to output this cycle E1122 06:38:56.413293 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:38:56.413337 3798194 memory.go:191] Add success. W1122 06:38:56.413371 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:38:56.413386 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:38:56.413390 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:38:56.417830 3798194 cpu.go:282] Add success. I1122 06:38:56.441224 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:38:56.441707 3798194 net.go:642] Add success. I1122 06:38:56.443096 3798194 net.go:778] primary dev: ETH0 I1122 06:38:56.443122 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:38:56.443136 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:38:57.451190 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:38:57.451392 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:38:57.451402 3798194 disk_worker.go:723] disk space is not compliant W1122 06:38:57.451406 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:38:57.459744 3798194 disk_worker.go:509] system disk:vda1 I1122 06:38:57.459789 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:38:58.446196 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:38:59.450404 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:38:59.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:38:59.450509 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:38:59.471014 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:39:02.676136 3798194 disk_info.go:125] begin check local disk info of client I1122 06:39:02.679966 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:39:02.679975 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000464980 0xc0004649c0] E1122 06:39:06.413348 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:39:06.413372 3798194 memory.go:184] no items to output this cycle I1122 06:39:06.413655 3798194 cpu.go:275] no items to output this cycle E1122 06:39:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:39:16.412263 3798194 memory.go:184] no items to output this cycle I1122 06:39:16.412442 3798194 cpu.go:275] no items to output this cycle I1122 06:39:17.651698 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:39:17.651706 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:39:26.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:39:26.412889 3798194 memory.go:191] Add success. I1122 06:39:26.417322 3798194 cpu.go:282] Add success. I1122 06:39:26.441735 3798194 net.go:642] Add success. I1122 06:39:26.443136 3798194 net.go:778] primary dev: ETH0 I1122 06:39:26.443152 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:39:26.443167 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:39:29.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:39:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:39:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:39:36.412788 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:39:36.412811 3798194 memory.go:184] no items to output this cycle I1122 06:39:36.412958 3798194 cpu.go:275] no items to output this cycle E1122 06:39:46.413172 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:39:46.413195 3798194 memory.go:184] no items to output this cycle I1122 06:39:46.413349 3798194 cpu.go:275] no items to output this cycle E1122 06:39:56.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:39:56.412895 3798194 memory.go:191] Add success. W1122 06:39:56.412930 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:39:56.412942 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:39:56.412945 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:39:56.417235 3798194 cpu.go:282] Add success. I1122 06:39:56.441610 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:39:56.442194 3798194 net.go:642] Add success. I1122 06:39:56.443689 3798194 net.go:778] primary dev: ETH0 I1122 06:39:56.443704 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:39:56.443717 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:39:57.451105 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:39:57.451340 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:39:57.451351 3798194 disk_worker.go:723] disk space is not compliant W1122 06:39:57.451354 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:39:57.459766 3798194 disk_worker.go:509] system disk:vda1 I1122 06:39:57.459808 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:39:58.445994 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:39:59.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:39:59.450089 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:39:59.450127 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:39:59.471648 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:40:02.680146 3798194 disk_info.go:125] begin check local disk info of client I1122 06:40:02.683946 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:40:02.683958 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000357340 0xc000357380] E1122 06:40:06.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:40:06.411974 3798194 memory.go:184] no items to output this cycle I1122 06:40:06.412123 3798194 cpu.go:275] no items to output this cycle E1122 06:40:16.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:40:16.413253 3798194 memory.go:184] no items to output this cycle I1122 06:40:16.413397 3798194 cpu.go:275] no items to output this cycle E1122 06:40:26.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:40:26.413277 3798194 memory.go:191] Add success. I1122 06:40:26.417522 3798194 cpu.go:282] Add success. I1122 06:40:26.442098 3798194 net.go:642] Add success. I1122 06:40:26.443591 3798194 net.go:778] primary dev: ETH0 I1122 06:40:26.443605 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:40:26.443617 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:40:29.450386 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:40:29.450463 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:40:29.450489 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:40:36.412977 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:40:36.413005 3798194 memory.go:184] no items to output this cycle I1122 06:40:36.413298 3798194 cpu.go:275] no items to output this cycle E1122 06:40:46.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:40:46.412252 3798194 memory.go:184] no items to output this cycle I1122 06:40:46.412399 3798194 cpu.go:275] no items to output this cycle E1122 06:40:56.412224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:40:56.412260 3798194 memory.go:191] Add success. W1122 06:40:56.412289 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:40:56.412301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:40:56.412304 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:40:56.416423 3798194 cpu.go:282] Add success. I1122 06:40:56.444696 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:40:56.445175 3798194 net.go:642] Add success. I1122 06:40:56.446679 3798194 net.go:778] primary dev: ETH0 I1122 06:40:56.446693 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:40:56.446705 3798194 net.go:659] Add eni metrics success, count: 7 E1122 06:40:57.450643 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 06:40:57.450657 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:40:57.450819 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:40:57.450876 3798194 disk_worker.go:723] disk space is not compliant W1122 06:40:57.450879 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:40:57.459124 3798194 disk_worker.go:509] system disk:vda1 I1122 06:40:57.459169 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 06:40:58.448500 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 06:40:58.448514 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 06:40:59.448963 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 06:40:59.450045 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:40:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:40:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:40:59.471689 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:41:02.684071 3798194 disk_info.go:125] begin check local disk info of client I1122 06:41:02.687887 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:41:02.687896 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b4f80 0xc0003b4fc0] E1122 06:41:06.412819 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:41:06.412853 3798194 memory.go:184] no items to output this cycle I1122 06:41:06.412996 3798194 cpu.go:275] no items to output this cycle E1122 06:41:16.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:41:16.413257 3798194 memory.go:184] no items to output this cycle I1122 06:41:16.413403 3798194 cpu.go:275] no items to output this cycle E1122 06:41:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:41:26.413269 3798194 memory.go:191] Add success. I1122 06:41:26.417457 3798194 cpu.go:282] Add success. I1122 06:41:26.442005 3798194 net.go:642] Add success. I1122 06:41:26.443502 3798194 net.go:778] primary dev: ETH0 I1122 06:41:26.443517 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:41:26.443531 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:41:29.449649 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:41:29.449718 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:41:29.449744 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:41:36.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:41:36.411986 3798194 memory.go:184] no items to output this cycle I1122 06:41:36.412155 3798194 cpu.go:275] no items to output this cycle E1122 06:41:46.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:41:46.413267 3798194 memory.go:184] no items to output this cycle I1122 06:41:46.413411 3798194 cpu.go:275] no items to output this cycle E1122 06:41:56.411929 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:41:56.411961 3798194 memory.go:191] Add success. W1122 06:41:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:41:56.412006 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:41:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:41:56.412359 3798194 cpu.go:282] Add success. I1122 06:41:56.441820 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:41:56.442522 3798194 net.go:642] Add success. I1122 06:41:56.446704 3798194 net.go:778] primary dev: ETH0 I1122 06:41:56.446720 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:41:56.446738 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:41:57.450623 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:41:57.450787 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:41:57.450855 3798194 disk_worker.go:723] disk space is not compliant W1122 06:41:57.450859 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:41:57.459383 3798194 disk_worker.go:509] system disk:vda1 I1122 06:41:57.459426 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:41:58.446013 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:41:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:41:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:41:59.450133 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:41:59.471654 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:42:02.687998 3798194 disk_info.go:125] begin check local disk info of client I1122 06:42:02.691789 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:42:02.691797 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000071500 0xc000071540] E1122 06:42:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:42:06.412255 3798194 memory.go:184] no items to output this cycle I1122 06:42:06.412410 3798194 cpu.go:275] no items to output this cycle E1122 06:42:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:42:16.411979 3798194 memory.go:184] no items to output this cycle I1122 06:42:16.412148 3798194 cpu.go:275] no items to output this cycle I1122 06:42:17.652214 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:42:17.652222 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:42:26.412280 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:42:26.412326 3798194 memory.go:191] Add success. I1122 06:42:26.416810 3798194 cpu.go:282] Add success. I1122 06:42:26.442023 3798194 net.go:642] Add success. I1122 06:42:26.443509 3798194 net.go:778] primary dev: ETH0 I1122 06:42:26.443535 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:42:26.443555 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:42:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:42:29.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:42:29.450495 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:42:36.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:42:36.413265 3798194 memory.go:184] no items to output this cycle I1122 06:42:36.413416 3798194 cpu.go:275] no items to output this cycle E1122 06:42:46.412334 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:42:46.412356 3798194 memory.go:184] no items to output this cycle I1122 06:42:46.412498 3798194 cpu.go:275] no items to output this cycle E1122 06:42:56.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:42:56.412899 3798194 memory.go:191] Add success. W1122 06:42:56.412930 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:42:56.412942 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:42:56.412945 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:42:56.416862 3798194 cpu.go:282] Add success. I1122 06:42:56.441250 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:42:56.441738 3798194 net.go:642] Add success. I1122 06:42:56.443146 3798194 net.go:778] primary dev: ETH0 I1122 06:42:56.443162 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:42:56.443175 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:42:57.456799 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:42:57.457004 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:42:57.457014 3798194 disk_worker.go:723] disk space is not compliant W1122 06:42:57.457017 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:42:57.468835 3798194 disk_worker.go:509] system disk:vda1 I1122 06:42:57.468879 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:42:58.445998 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:42:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:42:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:42:59.450140 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:42:59.471713 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:43:02.691889 3798194 disk_info.go:125] begin check local disk info of client I1122 06:43:02.695767 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:43:02.695777 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004180c0 0xc000418100] E1122 06:43:06.411251 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:43:06.411271 3798194 memory.go:184] no items to output this cycle I1122 06:43:06.411414 3798194 cpu.go:275] no items to output this cycle E1122 06:43:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:43:16.411982 3798194 memory.go:184] no items to output this cycle I1122 06:43:16.412145 3798194 cpu.go:275] no items to output this cycle E1122 06:43:26.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:43:26.413269 3798194 memory.go:191] Add success. I1122 06:43:26.417250 3798194 cpu.go:282] Add success. I1122 06:43:26.441755 3798194 net.go:642] Add success. I1122 06:43:26.443414 3798194 net.go:778] primary dev: ETH0 I1122 06:43:26.443428 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:43:26.443442 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:43:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:43:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:43:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:43:36.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:43:36.411989 3798194 memory.go:184] no items to output this cycle I1122 06:43:36.412144 3798194 cpu.go:275] no items to output this cycle E1122 06:43:46.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:43:46.412263 3798194 memory.go:184] no items to output this cycle I1122 06:43:46.412412 3798194 cpu.go:275] no items to output this cycle E1122 06:43:56.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:43:56.412893 3798194 memory.go:191] Add success. W1122 06:43:56.412921 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:43:56.412933 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:43:56.412937 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:43:56.417189 3798194 cpu.go:282] Add success. I1122 06:43:56.441562 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:43:56.442131 3798194 net.go:642] Add success. I1122 06:43:56.443556 3798194 net.go:778] primary dev: ETH0 I1122 06:43:56.443572 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:43:56.443584 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:43:57.450475 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:43:57.450632 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:43:57.450696 3798194 disk_worker.go:723] disk space is not compliant W1122 06:43:57.450700 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:43:57.458993 3798194 disk_worker.go:509] system disk:vda1 I1122 06:43:57.459031 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:43:58.445995 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:43:59.450007 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:43:59.450084 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:43:59.450119 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:43:59.471638 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:44:02.697144 3798194 disk_info.go:125] begin check local disk info of client I1122 06:44:02.701030 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:44:02.701039 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000380340 0xc000380380] E1122 06:44:06.412247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:44:06.412269 3798194 memory.go:184] no items to output this cycle I1122 06:44:06.412421 3798194 cpu.go:275] no items to output this cycle E1122 06:44:16.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:44:16.413248 3798194 memory.go:184] no items to output this cycle I1122 06:44:16.413406 3798194 cpu.go:275] no items to output this cycle E1122 06:44:26.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:44:26.413273 3798194 memory.go:191] Add success. I1122 06:44:26.417466 3798194 cpu.go:282] Add success. I1122 06:44:26.442024 3798194 net.go:642] Add success. I1122 06:44:26.443397 3798194 net.go:778] primary dev: ETH0 I1122 06:44:26.443412 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:44:26.443423 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:44:29.450576 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:44:29.450655 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:44:29.450681 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:44:36.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:44:36.412263 3798194 memory.go:184] no items to output this cycle I1122 06:44:36.412407 3798194 cpu.go:275] no items to output this cycle E1122 06:44:46.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:44:46.411993 3798194 memory.go:184] no items to output this cycle I1122 06:44:46.412149 3798194 cpu.go:275] no items to output this cycle E1122 06:44:56.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:44:56.411978 3798194 memory.go:191] Add success. W1122 06:44:56.412013 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:44:56.412024 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:44:56.412028 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:44:56.412413 3798194 cpu.go:282] Add success. I1122 06:44:56.441914 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:44:56.442465 3798194 net.go:642] Add success. I1122 06:44:56.446522 3798194 net.go:778] primary dev: ETH0 I1122 06:44:56.446537 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:44:56.446550 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:44:57.450908 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:44:57.451074 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:44:57.451152 3798194 disk_worker.go:723] disk space is not compliant W1122 06:44:57.451155 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:44:57.461316 3798194 disk_worker.go:509] system disk:vda1 I1122 06:44:57.461368 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:44:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:44:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:44:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:44:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:44:59.471637 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:45:02.702147 3798194 disk_info.go:125] begin check local disk info of client I1122 06:45:02.705996 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:45:02.706005 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000438540 0xc000438580] E1122 06:45:06.412809 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:45:06.412832 3798194 memory.go:184] no items to output this cycle I1122 06:45:06.412985 3798194 cpu.go:275] no items to output this cycle E1122 06:45:16.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:45:16.413249 3798194 memory.go:184] no items to output this cycle I1122 06:45:16.413398 3798194 cpu.go:275] no items to output this cycle I1122 06:45:17.652389 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:45:17.652397 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:45:26.411316 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:45:26.411350 3798194 memory.go:191] Add success. I1122 06:45:26.411676 3798194 cpu.go:282] Add success. I1122 06:45:26.441299 3798194 net.go:642] Add success. I1122 06:45:26.445677 3798194 net.go:778] primary dev: ETH0 I1122 06:45:26.445692 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:45:26.445707 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:45:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:45:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:45:29.450142 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:45:36.412833 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:45:36.412855 3798194 memory.go:184] no items to output this cycle I1122 06:45:36.413003 3798194 cpu.go:275] no items to output this cycle E1122 06:45:46.411969 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:45:46.411990 3798194 memory.go:184] no items to output this cycle I1122 06:45:46.412151 3798194 cpu.go:275] no items to output this cycle E1122 06:45:56.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:45:56.412262 3798194 memory.go:191] Add success. W1122 06:45:56.412288 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:45:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:45:56.412303 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:45:56.416670 3798194 cpu.go:282] Add success. I1122 06:45:56.443920 3798194 event_worker.go:152] Polling the log file for events... I1122 06:45:56.444164 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:45:56.444741 3798194 net.go:642] Add success. I1122 06:45:56.446249 3798194 net.go:778] primary dev: ETH0 I1122 06:45:56.446263 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:45:56.446275 3798194 net.go:659] Add eni metrics success, count: 7 E1122 06:45:57.450574 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 06:45:57.450588 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:45:57.450745 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:45:57.450811 3798194 disk_worker.go:723] disk space is not compliant W1122 06:45:57.450814 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:45:57.460025 3798194 disk_worker.go:509] system disk:vda1 I1122 06:45:57.460066 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 06:45:58.447817 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 06:45:58.447832 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 06:45:59.448611 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 06:45:59.449687 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:45:59.449739 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:45:59.449758 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:45:59.471267 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:46:02.707147 3798194 disk_info.go:125] begin check local disk info of client I1122 06:46:02.711000 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:46:02.711011 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000475840 0xc000475880] E1122 06:46:06.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:46:06.411974 3798194 memory.go:184] no items to output this cycle I1122 06:46:06.412134 3798194 cpu.go:275] no items to output this cycle E1122 06:46:16.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:46:16.412251 3798194 memory.go:184] no items to output this cycle I1122 06:46:16.412394 3798194 cpu.go:275] no items to output this cycle E1122 06:46:26.411933 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:46:26.411966 3798194 memory.go:191] Add success. I1122 06:46:26.412321 3798194 cpu.go:282] Add success. I1122 06:46:26.441957 3798194 net.go:642] Add success. I1122 06:46:26.445918 3798194 net.go:778] primary dev: ETH0 I1122 06:46:26.445936 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:46:26.445950 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:46:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:46:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:46:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:46:36.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:46:36.413257 3798194 memory.go:184] no items to output this cycle I1122 06:46:36.413413 3798194 cpu.go:275] no items to output this cycle E1122 06:46:46.413263 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:46:46.413286 3798194 memory.go:184] no items to output this cycle I1122 06:46:46.413433 3798194 cpu.go:275] no items to output this cycle E1122 06:46:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:46:56.411950 3798194 memory.go:191] Add success. W1122 06:46:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:46:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:46:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:46:56.412350 3798194 cpu.go:282] Add success. I1122 06:46:56.441862 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:46:56.442457 3798194 net.go:642] Add success. I1122 06:46:56.447823 3798194 net.go:778] primary dev: ETH0 I1122 06:46:56.447850 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:46:56.447864 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:46:57.450358 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:46:57.450569 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:46:57.450582 3798194 disk_worker.go:723] disk space is not compliant W1122 06:46:57.450585 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:46:57.458694 3798194 disk_worker.go:509] system disk:vda1 I1122 06:46:57.458736 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:46:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:46:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:46:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:46:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:46:59.471634 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:47:02.718142 3798194 disk_info.go:125] begin check local disk info of client I1122 06:47:02.735546 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:47:02.735557 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003bcc80 0xc0003bccc0] E1122 06:47:06.413126 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:47:06.413149 3798194 memory.go:184] no items to output this cycle I1122 06:47:06.413298 3798194 cpu.go:275] no items to output this cycle E1122 06:47:16.412454 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:47:16.412476 3798194 memory.go:184] no items to output this cycle I1122 06:47:16.412621 3798194 cpu.go:275] no items to output this cycle E1122 06:47:26.412846 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:47:26.412879 3798194 memory.go:191] Add success. I1122 06:47:26.417073 3798194 cpu.go:282] Add success. I1122 06:47:26.441303 3798194 net.go:642] Add success. I1122 06:47:26.442597 3798194 net.go:778] primary dev: ETH0 I1122 06:47:26.442611 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:47:26.442625 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:47:29.450494 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:47:29.450554 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:47:29.450583 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:47:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:47:36.413260 3798194 memory.go:184] no items to output this cycle I1122 06:47:36.413407 3798194 cpu.go:275] no items to output this cycle E1122 06:47:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:47:46.411973 3798194 memory.go:184] no items to output this cycle I1122 06:47:46.412139 3798194 cpu.go:275] no items to output this cycle E1122 06:47:56.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:47:56.413281 3798194 memory.go:191] Add success. W1122 06:47:56.413314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:47:56.413326 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:47:56.413329 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:47:56.417368 3798194 cpu.go:282] Add success. I1122 06:47:56.441783 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:47:56.442320 3798194 net.go:642] Add success. I1122 06:47:56.444268 3798194 net.go:778] primary dev: ETH0 I1122 06:47:56.444283 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:47:56.444295 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:47:57.452126 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:47:57.452298 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:47:57.452355 3798194 disk_worker.go:723] disk space is not compliant W1122 06:47:57.452358 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:47:57.464187 3798194 disk_worker.go:509] system disk:vda1 I1122 06:47:57.464237 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:47:58.446431 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:47:59.450528 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:47:59.450607 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:47:59.450633 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:47:59.471129 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:48:02.735661 3798194 disk_info.go:125] begin check local disk info of client I1122 06:48:02.739553 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:48:02.739562 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002f4840 0xc0002f4880] E1122 06:48:06.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:48:06.412252 3798194 memory.go:184] no items to output this cycle I1122 06:48:06.412389 3798194 cpu.go:275] no items to output this cycle E1122 06:48:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:48:16.411976 3798194 memory.go:184] no items to output this cycle I1122 06:48:16.412145 3798194 cpu.go:275] no items to output this cycle I1122 06:48:17.652557 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:48:17.652566 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:48:26.411222 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:48:26.411258 3798194 memory.go:191] Add success. I1122 06:48:26.411593 3798194 cpu.go:282] Add success. I1122 06:48:26.442233 3798194 net.go:642] Add success. I1122 06:48:26.446007 3798194 net.go:778] primary dev: ETH0 I1122 06:48:26.446024 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:48:26.446041 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:48:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:48:29.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:48:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:48:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:48:36.412257 3798194 memory.go:184] no items to output this cycle I1122 06:48:36.412404 3798194 cpu.go:275] no items to output this cycle E1122 06:48:46.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:48:46.411970 3798194 memory.go:184] no items to output this cycle I1122 06:48:46.412128 3798194 cpu.go:275] no items to output this cycle E1122 06:48:56.413272 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:48:56.413312 3798194 memory.go:191] Add success. W1122 06:48:56.413347 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:48:56.413359 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:48:56.413363 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:48:56.417617 3798194 cpu.go:282] Add success. I1122 06:48:56.441161 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:48:56.441650 3798194 net.go:642] Add success. I1122 06:48:56.443121 3798194 net.go:778] primary dev: ETH0 I1122 06:48:56.443138 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:48:56.443151 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:48:57.455686 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:48:57.455853 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:48:57.455908 3798194 disk_worker.go:723] disk space is not compliant W1122 06:48:57.455912 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:48:57.467698 3798194 disk_worker.go:509] system disk:vda1 I1122 06:48:57.467741 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:48:58.446133 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:48:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:48:59.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:48:59.450498 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:48:59.471020 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:49:02.740146 3798194 disk_info.go:125] begin check local disk info of client I1122 06:49:02.743984 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:49:02.743994 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0180 0xc0002e01c0] E1122 06:49:06.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:49:06.412253 3798194 memory.go:184] no items to output this cycle I1122 06:49:06.412394 3798194 cpu.go:275] no items to output this cycle E1122 06:49:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:49:16.413264 3798194 memory.go:184] no items to output this cycle I1122 06:49:16.413422 3798194 cpu.go:275] no items to output this cycle E1122 06:49:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:49:26.413267 3798194 memory.go:191] Add success. I1122 06:49:26.417707 3798194 cpu.go:282] Add success. I1122 06:49:26.442288 3798194 net.go:642] Add success. I1122 06:49:26.443844 3798194 net.go:778] primary dev: ETH0 I1122 06:49:26.443860 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:49:26.443872 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:49:29.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:49:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:49:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:49:36.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:49:36.412878 3798194 memory.go:184] no items to output this cycle I1122 06:49:36.413022 3798194 cpu.go:275] no items to output this cycle E1122 06:49:46.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:49:46.412258 3798194 memory.go:184] no items to output this cycle I1122 06:49:46.412408 3798194 cpu.go:275] no items to output this cycle E1122 06:49:56.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:49:56.413270 3798194 memory.go:191] Add success. W1122 06:49:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:49:56.413316 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:49:56.413320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:49:56.417825 3798194 cpu.go:282] Add success. I1122 06:49:56.441228 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:49:56.441726 3798194 net.go:642] Add success. I1122 06:49:56.443231 3798194 net.go:778] primary dev: ETH0 I1122 06:49:56.443246 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:49:56.443258 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:49:57.450821 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:49:57.451030 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:49:57.451041 3798194 disk_worker.go:723] disk space is not compliant W1122 06:49:57.451044 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:49:57.460512 3798194 disk_worker.go:509] system disk:vda1 I1122 06:49:57.460550 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:49:58.446372 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:49:59.450385 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:49:59.450466 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:49:59.450493 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:49:59.471023 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:50:02.744142 3798194 disk_info.go:125] begin check local disk info of client I1122 06:50:02.748136 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:50:02.748147 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ff840 0xc0003ff880] E1122 06:50:06.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:50:06.413262 3798194 memory.go:184] no items to output this cycle I1122 06:50:06.413420 3798194 cpu.go:275] no items to output this cycle E1122 06:50:16.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:50:16.413269 3798194 memory.go:184] no items to output this cycle I1122 06:50:16.413414 3798194 cpu.go:275] no items to output this cycle E1122 06:50:26.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:50:26.413275 3798194 memory.go:191] Add success. I1122 06:50:26.417732 3798194 cpu.go:282] Add success. I1122 06:50:26.441258 3798194 net.go:642] Add success. I1122 06:50:26.442644 3798194 net.go:778] primary dev: ETH0 I1122 06:50:26.442662 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:50:26.442675 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:50:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:50:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:50:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:50:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:50:36.412262 3798194 memory.go:184] no items to output this cycle I1122 06:50:36.412404 3798194 cpu.go:275] no items to output this cycle E1122 06:50:46.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:50:46.413255 3798194 memory.go:184] no items to output this cycle I1122 06:50:46.413399 3798194 cpu.go:275] no items to output this cycle E1122 06:50:56.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:50:56.412891 3798194 memory.go:191] Add success. W1122 06:50:56.412923 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:50:56.412934 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:50:56.412938 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:50:56.418076 3798194 cpu.go:282] Add success. I1122 06:50:56.442953 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:50:56.443566 3798194 net.go:642] Add success. I1122 06:50:56.445541 3798194 net.go:778] primary dev: ETH0 I1122 06:50:56.445557 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:50:56.445570 3798194 net.go:659] Add eni metrics success, count: 7 E1122 06:50:57.446308 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 06:50:57.446323 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:50:57.446519 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:50:57.446578 3798194 disk_worker.go:723] disk space is not compliant W1122 06:50:57.446581 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:50:57.449336 3798194 disk_worker.go:509] system disk:vda1 I1122 06:50:57.449387 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 06:50:58.448537 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 06:50:58.448552 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 06:50:59.448995 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 06:50:59.450069 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:50:59.450132 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:50:59.450152 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:50:59.471687 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:51:02.749138 3798194 disk_info.go:125] begin check local disk info of client I1122 06:51:02.753005 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:51:02.753015 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d24c0 0xc0003d2500] E1122 06:51:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:51:06.411978 3798194 memory.go:184] no items to output this cycle I1122 06:51:06.412127 3798194 cpu.go:275] no items to output this cycle E1122 06:51:16.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:51:16.411988 3798194 memory.go:184] no items to output this cycle I1122 06:51:16.412150 3798194 cpu.go:275] no items to output this cycle I1122 06:51:17.652724 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:51:17.652732 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:51:26.411579 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:51:26.411614 3798194 memory.go:191] Add success. I1122 06:51:26.412213 3798194 cpu.go:282] Add success. I1122 06:51:26.441299 3798194 net.go:642] Add success. I1122 06:51:26.445180 3798194 net.go:778] primary dev: ETH0 I1122 06:51:26.445194 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:51:26.445208 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:51:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:51:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:51:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:51:36.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:51:36.411974 3798194 memory.go:184] no items to output this cycle I1122 06:51:36.412133 3798194 cpu.go:275] no items to output this cycle E1122 06:51:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:51:46.411971 3798194 memory.go:184] no items to output this cycle I1122 06:51:46.412127 3798194 cpu.go:275] no items to output this cycle E1122 06:51:56.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:51:56.413269 3798194 memory.go:191] Add success. W1122 06:51:56.413301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:51:56.413313 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:51:56.413318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:51:56.417723 3798194 cpu.go:282] Add success. I1122 06:51:56.441269 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:51:56.441747 3798194 net.go:642] Add success. I1122 06:51:56.443244 3798194 net.go:778] primary dev: ETH0 I1122 06:51:56.443260 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:51:56.443273 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:51:57.450437 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:51:57.450598 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:51:57.450654 3798194 disk_worker.go:723] disk space is not compliant W1122 06:51:57.450657 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:51:57.458932 3798194 disk_worker.go:509] system disk:vda1 I1122 06:51:57.458974 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:51:58.446334 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:51:59.450436 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:51:59.450513 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:51:59.450539 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:51:59.471059 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:52:02.753133 3798194 disk_info.go:125] begin check local disk info of client I1122 06:52:02.756995 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:52:02.757003 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d26c0 0xc0003d2700] E1122 06:52:06.413513 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:52:06.413534 3798194 memory.go:184] no items to output this cycle I1122 06:52:06.413686 3798194 cpu.go:275] no items to output this cycle E1122 06:52:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:52:16.413256 3798194 memory.go:184] no items to output this cycle I1122 06:52:16.413402 3798194 cpu.go:275] no items to output this cycle E1122 06:52:26.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:52:26.412267 3798194 memory.go:191] Add success. I1122 06:52:26.416556 3798194 cpu.go:282] Add success. I1122 06:52:26.442123 3798194 net.go:642] Add success. I1122 06:52:26.443784 3798194 net.go:778] primary dev: ETH0 I1122 06:52:26.443798 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:52:26.443811 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:52:29.450053 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:52:29.450144 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:52:29.450169 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:52:36.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:52:36.413252 3798194 memory.go:184] no items to output this cycle I1122 06:52:36.413398 3798194 cpu.go:275] no items to output this cycle E1122 06:52:46.413404 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:52:46.413425 3798194 memory.go:184] no items to output this cycle I1122 06:52:46.413699 3798194 cpu.go:275] no items to output this cycle E1122 06:52:56.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:52:56.413270 3798194 memory.go:191] Add success. W1122 06:52:56.413310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:52:56.413322 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:52:56.413325 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:52:56.417933 3798194 cpu.go:282] Add success. I1122 06:52:56.441347 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:52:56.441844 3798194 net.go:642] Add success. I1122 06:52:56.443698 3798194 net.go:778] primary dev: ETH0 I1122 06:52:56.443713 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:52:56.443727 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:52:57.446274 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:52:57.446542 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:52:57.446554 3798194 disk_worker.go:723] disk space is not compliant W1122 06:52:57.446558 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:52:57.451574 3798194 disk_worker.go:509] system disk:vda1 I1122 06:52:57.451612 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:52:58.446387 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:52:59.450494 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:52:59.450575 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:52:59.450604 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:52:59.471128 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:53:02.757117 3798194 disk_info.go:125] begin check local disk info of client I1122 06:53:02.760985 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:53:02.760995 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c2680 0xc0003c26c0] E1122 06:53:06.412802 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:53:06.412823 3798194 memory.go:184] no items to output this cycle I1122 06:53:06.412968 3798194 cpu.go:275] no items to output this cycle E1122 06:53:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:53:16.411977 3798194 memory.go:184] no items to output this cycle I1122 06:53:16.412141 3798194 cpu.go:275] no items to output this cycle E1122 06:53:26.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:53:26.412265 3798194 memory.go:191] Add success. I1122 06:53:26.416768 3798194 cpu.go:282] Add success. I1122 06:53:26.441285 3798194 net.go:642] Add success. I1122 06:53:26.442872 3798194 net.go:778] primary dev: ETH0 I1122 06:53:26.442886 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:53:26.442899 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:53:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:53:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:53:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:53:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:53:36.411979 3798194 memory.go:184] no items to output this cycle I1122 06:53:36.412131 3798194 cpu.go:275] no items to output this cycle E1122 06:53:46.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:53:46.411972 3798194 memory.go:184] no items to output this cycle I1122 06:53:46.412130 3798194 cpu.go:275] no items to output this cycle E1122 06:53:56.413249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:53:56.413290 3798194 memory.go:191] Add success. W1122 06:53:56.413323 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:53:56.413335 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:53:56.413339 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:53:56.417543 3798194 cpu.go:282] Add success. I1122 06:53:56.440922 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:53:56.441503 3798194 net.go:642] Add success. I1122 06:53:56.442964 3798194 net.go:778] primary dev: ETH0 I1122 06:53:56.442978 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:53:56.442992 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:53:57.449529 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:53:57.449790 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:53:57.449803 3798194 disk_worker.go:723] disk space is not compliant W1122 06:53:57.449807 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:53:57.451730 3798194 disk_worker.go:509] system disk:vda1 I1122 06:53:57.451786 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:53:58.445989 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:53:59.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:53:59.450130 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:53:59.450162 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:53:59.470681 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:54:02.761101 3798194 disk_info.go:125] begin check local disk info of client I1122 06:54:02.765533 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:54:02.765544 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a49c0 0xc0004a4a00] E1122 06:54:06.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:54:06.411987 3798194 memory.go:184] no items to output this cycle I1122 06:54:06.412152 3798194 cpu.go:275] no items to output this cycle E1122 06:54:16.412248 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:54:16.412272 3798194 memory.go:184] no items to output this cycle I1122 06:54:16.412417 3798194 cpu.go:275] no items to output this cycle I1122 06:54:17.653340 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:54:17.653349 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:54:26.411221 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:54:26.411253 3798194 memory.go:191] Add success. I1122 06:54:26.411589 3798194 cpu.go:282] Add success. I1122 06:54:26.442269 3798194 net.go:642] Add success. I1122 06:54:26.446423 3798194 net.go:778] primary dev: ETH0 I1122 06:54:26.446440 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:54:26.446454 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:54:29.449953 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:54:29.450036 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:54:29.450062 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:54:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:54:36.412256 3798194 memory.go:184] no items to output this cycle I1122 06:54:36.412413 3798194 cpu.go:275] no items to output this cycle E1122 06:54:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:54:46.411977 3798194 memory.go:184] no items to output this cycle I1122 06:54:46.412137 3798194 cpu.go:275] no items to output this cycle E1122 06:54:56.411926 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:54:56.411960 3798194 memory.go:191] Add success. W1122 06:54:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:54:56.412003 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:54:56.412006 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:54:56.412352 3798194 cpu.go:282] Add success. I1122 06:54:56.441851 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:54:56.442518 3798194 net.go:642] Add success. I1122 06:54:56.446553 3798194 net.go:778] primary dev: ETH0 I1122 06:54:56.446569 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:54:56.446581 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:54:57.451514 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:54:57.451720 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:54:57.451731 3798194 disk_worker.go:723] disk space is not compliant W1122 06:54:57.451734 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:54:57.461803 3798194 disk_worker.go:509] system disk:vda1 I1122 06:54:57.461846 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:54:58.445997 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:54:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:54:59.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:54:59.450500 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:54:59.471024 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:55:02.765638 3798194 disk_info.go:125] begin check local disk info of client I1122 06:55:02.769503 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:55:02.769512 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b43c0 0xc0003b4400] E1122 06:55:06.412106 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:55:06.412142 3798194 memory.go:184] no items to output this cycle I1122 06:55:06.412287 3798194 cpu.go:275] no items to output this cycle E1122 06:55:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:55:16.413258 3798194 memory.go:184] no items to output this cycle I1122 06:55:16.413404 3798194 cpu.go:275] no items to output this cycle E1122 06:55:26.412033 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:55:26.412068 3798194 memory.go:191] Add success. I1122 06:55:26.412633 3798194 cpu.go:282] Add success. I1122 06:55:26.441298 3798194 net.go:642] Add success. I1122 06:55:26.445353 3798194 net.go:778] primary dev: ETH0 I1122 06:55:26.445367 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:55:26.445388 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:55:29.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:55:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:55:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:55:36.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:55:36.412879 3798194 memory.go:184] no items to output this cycle I1122 06:55:36.413026 3798194 cpu.go:275] no items to output this cycle E1122 06:55:46.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:55:46.412258 3798194 memory.go:184] no items to output this cycle I1122 06:55:46.412399 3798194 cpu.go:275] no items to output this cycle E1122 06:55:56.412284 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:55:56.412326 3798194 memory.go:191] Add success. W1122 06:55:56.412358 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:55:56.412373 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:55:56.412378 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:55:56.416934 3798194 cpu.go:282] Add success. I1122 06:55:56.444552 3798194 event_worker.go:152] Polling the log file for events... I1122 06:55:56.444869 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:55:56.445750 3798194 net.go:642] Add success. I1122 06:55:56.447377 3798194 net.go:778] primary dev: ETH0 I1122 06:55:56.447400 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:55:56.447419 3798194 net.go:659] Add eni metrics success, count: 7 E1122 06:55:57.451087 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 06:55:57.451101 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:55:57.451271 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:55:57.451322 3798194 disk_worker.go:723] disk space is not compliant W1122 06:55:57.451325 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:55:57.459820 3798194 disk_worker.go:509] system disk:vda1 I1122 06:55:57.459864 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 06:55:58.448211 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 06:55:58.448226 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 06:55:59.448754 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 06:55:59.449829 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:55:59.449882 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:55:59.449904 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:55:59.471442 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:56:02.770152 3798194 disk_info.go:125] begin check local disk info of client I1122 06:56:02.773995 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:56:02.774004 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00027cdc0 0xc00027ce00] E1122 06:56:06.412994 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:56:06.413026 3798194 memory.go:184] no items to output this cycle I1122 06:56:06.413187 3798194 cpu.go:275] no items to output this cycle E1122 06:56:16.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:56:16.413255 3798194 memory.go:184] no items to output this cycle I1122 06:56:16.413411 3798194 cpu.go:275] no items to output this cycle E1122 06:56:26.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:56:26.412268 3798194 memory.go:191] Add success. I1122 06:56:26.416627 3798194 cpu.go:282] Add success. I1122 06:56:26.442193 3798194 net.go:642] Add success. I1122 06:56:26.443713 3798194 net.go:778] primary dev: ETH0 I1122 06:56:26.443727 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:56:26.443742 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:56:29.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:56:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:56:29.450502 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:56:36.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:56:36.413262 3798194 memory.go:184] no items to output this cycle I1122 06:56:36.413408 3798194 cpu.go:275] no items to output this cycle E1122 06:56:46.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:56:46.413267 3798194 memory.go:184] no items to output this cycle I1122 06:56:46.413413 3798194 cpu.go:275] no items to output this cycle E1122 06:56:56.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:56:56.413279 3798194 memory.go:191] Add success. W1122 06:56:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:56:56.413324 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:56:56.413328 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:56:56.417676 3798194 cpu.go:282] Add success. I1122 06:56:56.441070 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:56:56.441646 3798194 net.go:642] Add success. I1122 06:56:56.443158 3798194 net.go:778] primary dev: ETH0 I1122 06:56:56.443172 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:56:56.443184 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:56:57.451378 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:56:57.451585 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:56:57.451596 3798194 disk_worker.go:723] disk space is not compliant W1122 06:56:57.451600 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:56:57.460886 3798194 disk_worker.go:509] system disk:vda1 I1122 06:56:57.460929 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:56:58.446508 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:56:59.450100 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:56:59.450184 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:56:59.450212 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:56:59.470688 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:57:02.775144 3798194 disk_info.go:125] begin check local disk info of client I1122 06:57:02.778990 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:57:02.779001 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0700 0xc0002e0740] E1122 06:57:06.412865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:57:06.412886 3798194 memory.go:184] no items to output this cycle I1122 06:57:06.413037 3798194 cpu.go:275] no items to output this cycle E1122 06:57:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:57:16.411973 3798194 memory.go:184] no items to output this cycle I1122 06:57:16.412134 3798194 cpu.go:275] no items to output this cycle I1122 06:57:17.653513 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 06:57:17.653521 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 06:57:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:57:26.413273 3798194 memory.go:191] Add success. I1122 06:57:26.417446 3798194 cpu.go:282] Add success. I1122 06:57:26.441989 3798194 net.go:642] Add success. I1122 06:57:26.443511 3798194 net.go:778] primary dev: ETH0 I1122 06:57:26.443525 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:57:26.443538 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:57:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:57:29.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:57:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:57:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:57:36.412258 3798194 memory.go:184] no items to output this cycle I1122 06:57:36.412402 3798194 cpu.go:275] no items to output this cycle E1122 06:57:46.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:57:46.413261 3798194 memory.go:184] no items to output this cycle I1122 06:57:46.413410 3798194 cpu.go:275] no items to output this cycle E1122 06:57:56.413103 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:57:56.413148 3798194 memory.go:191] Add success. W1122 06:57:56.413183 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:57:56.413195 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:57:56.413198 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:57:56.417444 3798194 cpu.go:282] Add success. I1122 06:57:56.441792 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:57:56.442291 3798194 net.go:642] Add success. I1122 06:57:56.443730 3798194 net.go:778] primary dev: ETH0 I1122 06:57:56.443745 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:57:56.443757 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:57:57.451026 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:57:57.451224 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:57:57.451279 3798194 disk_worker.go:723] disk space is not compliant W1122 06:57:57.451282 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:57:57.460449 3798194 disk_worker.go:509] system disk:vda1 I1122 06:57:57.460487 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:57:58.446374 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:57:59.450475 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:57:59.450554 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:57:59.450580 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:57:59.471107 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:58:02.779143 3798194 disk_info.go:125] begin check local disk info of client I1122 06:58:02.782950 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:58:02.782959 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000403840 0xc000403880] E1122 06:58:06.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:58:06.412884 3798194 memory.go:184] no items to output this cycle I1122 06:58:06.413028 3798194 cpu.go:275] no items to output this cycle E1122 06:58:16.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:58:16.411982 3798194 memory.go:184] no items to output this cycle I1122 06:58:16.412151 3798194 cpu.go:275] no items to output this cycle E1122 06:58:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:58:26.413266 3798194 memory.go:191] Add success. I1122 06:58:26.417622 3798194 cpu.go:282] Add success. I1122 06:58:26.442180 3798194 net.go:642] Add success. I1122 06:58:26.443619 3798194 net.go:778] primary dev: ETH0 I1122 06:58:26.443634 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:58:26.443647 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:58:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:58:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:58:29.450503 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:58:36.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:58:36.413257 3798194 memory.go:184] no items to output this cycle I1122 06:58:36.413396 3798194 cpu.go:275] no items to output this cycle E1122 06:58:46.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:58:46.411986 3798194 memory.go:184] no items to output this cycle I1122 06:58:46.412147 3798194 cpu.go:275] no items to output this cycle E1122 06:58:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:58:56.411954 3798194 memory.go:191] Add success. W1122 06:58:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:58:56.411998 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:58:56.412002 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:58:56.412352 3798194 cpu.go:282] Add success. I1122 06:58:56.441802 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:58:56.442323 3798194 net.go:642] Add success. I1122 06:58:56.446216 3798194 net.go:778] primary dev: ETH0 I1122 06:58:56.446231 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:58:56.446242 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:58:57.450802 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:58:57.451029 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:58:57.451041 3798194 disk_worker.go:723] disk space is not compliant W1122 06:58:57.451044 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:58:57.459397 3798194 disk_worker.go:509] system disk:vda1 I1122 06:58:57.459441 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:58:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:58:59.450117 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:58:59.450186 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:58:59.450211 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:58:59.471724 3798194 disk_local_worker.go:436] Get disk info: [] I1122 06:59:02.784137 3798194 disk_info.go:125] begin check local disk info of client I1122 06:59:02.788089 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 06:59:02.788099 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f6300 0xc0003f6340] E1122 06:59:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:59:06.412259 3798194 memory.go:184] no items to output this cycle I1122 06:59:06.412406 3798194 cpu.go:275] no items to output this cycle E1122 06:59:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:59:16.411978 3798194 memory.go:184] no items to output this cycle I1122 06:59:16.412139 3798194 cpu.go:275] no items to output this cycle E1122 06:59:26.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:59:26.411944 3798194 memory.go:191] Add success. I1122 06:59:26.412492 3798194 cpu.go:282] Add success. I1122 06:59:26.441305 3798194 net.go:642] Add success. I1122 06:59:26.445138 3798194 net.go:778] primary dev: ETH0 I1122 06:59:26.445153 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:59:26.445165 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:59:29.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:59:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:59:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 06:59:36.413085 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:59:36.413108 3798194 memory.go:184] no items to output this cycle I1122 06:59:36.413273 3798194 cpu.go:275] no items to output this cycle E1122 06:59:46.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:59:46.412255 3798194 memory.go:184] no items to output this cycle I1122 06:59:46.412403 3798194 cpu.go:275] no items to output this cycle E1122 06:59:56.413004 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 06:59:56.413038 3798194 memory.go:191] Add success. W1122 06:59:56.413067 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 06:59:56.413079 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 06:59:56.413083 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 06:59:56.417362 3798194 cpu.go:282] Add success. I1122 06:59:56.441723 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 06:59:56.442222 3798194 net.go:642] Add success. I1122 06:59:56.443656 3798194 net.go:778] primary dev: ETH0 I1122 06:59:56.443670 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 06:59:56.443681 3798194 net.go:659] Add eni metrics success, count: 7 I1122 06:59:57.450729 3798194 custom_config.go:106] query custom config with name: gpu W1122 06:59:57.450924 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 06:59:57.450934 3798194 disk_worker.go:723] disk space is not compliant W1122 06:59:57.450938 3798194 disk_worker.go:743] disk inode is not compliant I1122 06:59:57.459331 3798194 disk_worker.go:509] system disk:vda1 I1122 06:59:57.459375 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 06:59:58.445703 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 06:59:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 06:59:59.450134 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 06:59:59.450176 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 06:59:59.470693 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:00:02.789454 3798194 disk_info.go:125] begin check local disk info of client I1122 07:00:02.793360 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:00:02.793379 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00024e840 0xc00024e880] E1122 07:00:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:00:06.411987 3798194 memory.go:184] no items to output this cycle I1122 07:00:06.412145 3798194 cpu.go:275] no items to output this cycle E1122 07:00:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:00:16.411971 3798194 memory.go:184] no items to output this cycle I1122 07:00:16.412122 3798194 cpu.go:275] no items to output this cycle I1122 07:00:17.653690 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:00:17.653699 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:00:26.412869 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:00:26.412906 3798194 memory.go:191] Add success. I1122 07:00:26.416940 3798194 cpu.go:282] Add success. I1122 07:00:26.441469 3798194 net.go:642] Add success. I1122 07:00:26.442869 3798194 net.go:778] primary dev: ETH0 I1122 07:00:26.442882 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:00:26.442895 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:00:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:00:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:00:29.450511 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:00:36.412445 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:00:36.412468 3798194 memory.go:184] no items to output this cycle I1122 07:00:36.412617 3798194 cpu.go:275] no items to output this cycle E1122 07:00:46.412003 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:00:46.412024 3798194 memory.go:184] no items to output this cycle I1122 07:00:46.412183 3798194 cpu.go:275] no items to output this cycle E1122 07:00:56.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:00:56.412278 3798194 memory.go:191] Add success. W1122 07:00:56.412305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:00:56.412315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:00:56.412319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:00:56.416745 3798194 cpu.go:282] Add success. I1122 07:00:56.444020 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:00:56.444519 3798194 net.go:642] Add success. I1122 07:00:56.446075 3798194 net.go:778] primary dev: ETH0 I1122 07:00:56.446092 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:00:56.446105 3798194 net.go:659] Add eni metrics success, count: 7 E1122 07:00:57.450577 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 07:00:57.450591 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:00:57.450779 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:00:57.450789 3798194 disk_worker.go:723] disk space is not compliant W1122 07:00:57.450792 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:00:57.459195 3798194 disk_worker.go:509] system disk:vda1 I1122 07:00:57.459237 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 07:00:58.448203 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 07:00:58.448217 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 07:00:59.448804 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 07:00:59.449878 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:00:59.449930 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:00:59.449949 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:00:59.471492 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:01:02.793489 3798194 disk_info.go:125] begin check local disk info of client I1122 07:01:02.797398 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:01:02.797407 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e0300 0xc0002e0340] E1122 07:01:06.413592 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:01:06.413616 3798194 memory.go:184] no items to output this cycle I1122 07:01:06.413758 3798194 cpu.go:275] no items to output this cycle E1122 07:01:16.413156 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:01:16.413181 3798194 memory.go:184] no items to output this cycle I1122 07:01:16.413334 3798194 cpu.go:275] no items to output this cycle E1122 07:01:26.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:01:26.413271 3798194 memory.go:191] Add success. I1122 07:01:26.417577 3798194 cpu.go:282] Add success. I1122 07:01:26.442162 3798194 net.go:642] Add success. I1122 07:01:26.443498 3798194 net.go:778] primary dev: ETH0 I1122 07:01:26.443512 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:01:26.443527 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:01:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:01:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:01:29.450506 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:01:36.412512 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:01:36.412541 3798194 memory.go:184] no items to output this cycle I1122 07:01:36.412742 3798194 cpu.go:275] no items to output this cycle E1122 07:01:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:01:46.412255 3798194 memory.go:184] no items to output this cycle I1122 07:01:46.412398 3798194 cpu.go:275] no items to output this cycle E1122 07:01:56.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:01:56.412896 3798194 memory.go:191] Add success. W1122 07:01:56.412924 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:01:56.412936 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:01:56.412940 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:01:56.417244 3798194 cpu.go:282] Add success. I1122 07:01:56.441638 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:01:56.442135 3798194 net.go:642] Add success. I1122 07:01:56.443659 3798194 net.go:778] primary dev: ETH0 I1122 07:01:56.443673 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:01:56.443687 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:01:57.450932 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:01:57.451089 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:01:57.451161 3798194 disk_worker.go:723] disk space is not compliant W1122 07:01:57.451164 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:01:57.460325 3798194 disk_worker.go:509] system disk:vda1 I1122 07:01:57.460364 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:01:58.446369 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:01:59.450470 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:01:59.450554 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:01:59.450580 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:01:59.471107 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:02:02.798145 3798194 disk_info.go:125] begin check local disk info of client I1122 07:02:02.802010 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:02:02.802019 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00045f840 0xc00045f880] E1122 07:02:06.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:02:06.412882 3798194 memory.go:184] no items to output this cycle I1122 07:02:06.413027 3798194 cpu.go:275] no items to output this cycle E1122 07:02:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:02:16.411974 3798194 memory.go:184] no items to output this cycle I1122 07:02:16.412130 3798194 cpu.go:275] no items to output this cycle E1122 07:02:26.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:02:26.413272 3798194 memory.go:191] Add success. I1122 07:02:26.417447 3798194 cpu.go:282] Add success. I1122 07:02:26.442007 3798194 net.go:642] Add success. I1122 07:02:26.443561 3798194 net.go:778] primary dev: ETH0 I1122 07:02:26.443577 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:02:26.443591 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:02:29.450450 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:02:29.450529 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:02:29.450559 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:02:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:02:36.413260 3798194 memory.go:184] no items to output this cycle I1122 07:02:36.413400 3798194 cpu.go:275] no items to output this cycle E1122 07:02:46.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:02:46.411991 3798194 memory.go:184] no items to output this cycle I1122 07:02:46.412156 3798194 cpu.go:275] no items to output this cycle E1122 07:02:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:02:56.412268 3798194 memory.go:191] Add success. W1122 07:02:56.412297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:02:56.412309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:02:56.412312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:02:56.416527 3798194 cpu.go:282] Add success. I1122 07:02:56.440939 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:02:56.441516 3798194 net.go:642] Add success. I1122 07:02:56.443006 3798194 net.go:778] primary dev: ETH0 I1122 07:02:56.443020 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:02:56.443035 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:02:57.448987 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:02:57.449208 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:02:57.449269 3798194 disk_worker.go:723] disk space is not compliant W1122 07:02:57.449273 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:02:57.451060 3798194 disk_worker.go:509] system disk:vda1 I1122 07:02:57.451103 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:02:58.445983 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:02:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:02:59.450127 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:02:59.450154 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:02:59.471702 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:03:02.803139 3798194 disk_info.go:125] begin check local disk info of client I1122 07:03:02.807077 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:03:02.807086 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b0300 0xc0003b0340] E1122 07:03:06.413105 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:03:06.413136 3798194 memory.go:184] no items to output this cycle I1122 07:03:06.413279 3798194 cpu.go:275] no items to output this cycle E1122 07:03:16.412933 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:03:16.412956 3798194 memory.go:184] no items to output this cycle I1122 07:03:16.413102 3798194 cpu.go:275] no items to output this cycle I1122 07:03:17.654206 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:03:17.654214 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:03:26.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:03:26.412264 3798194 memory.go:191] Add success. I1122 07:03:26.416820 3798194 cpu.go:282] Add success. I1122 07:03:26.441285 3798194 net.go:642] Add success. I1122 07:03:26.442824 3798194 net.go:778] primary dev: ETH0 I1122 07:03:26.442838 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:03:26.442850 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:03:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:03:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:03:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:03:36.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:03:36.411968 3798194 memory.go:184] no items to output this cycle I1122 07:03:36.412135 3798194 cpu.go:275] no items to output this cycle E1122 07:03:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:03:46.411983 3798194 memory.go:184] no items to output this cycle I1122 07:03:46.412141 3798194 cpu.go:275] no items to output this cycle E1122 07:03:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:03:56.411958 3798194 memory.go:191] Add success. W1122 07:03:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:03:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:03:56.412000 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:03:56.412349 3798194 cpu.go:282] Add success. I1122 07:03:56.441789 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:03:56.442369 3798194 net.go:642] Add success. I1122 07:03:56.446353 3798194 net.go:778] primary dev: ETH0 I1122 07:03:56.446367 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:03:56.446378 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:03:57.455924 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:03:57.456223 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:03:57.456238 3798194 disk_worker.go:723] disk space is not compliant W1122 07:03:57.456243 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:03:57.467618 3798194 disk_worker.go:509] system disk:vda1 I1122 07:03:57.467685 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:03:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:03:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:03:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:03:59.450504 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:03:59.471048 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:04:02.808182 3798194 disk_info.go:125] begin check local disk info of client I1122 07:04:02.818455 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:04:02.818467 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002fe840 0xc0002fe880] E1122 07:04:06.412378 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:04:06.412399 3798194 memory.go:184] no items to output this cycle I1122 07:04:06.412538 3798194 cpu.go:275] no items to output this cycle E1122 07:04:16.412900 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:04:16.412924 3798194 memory.go:184] no items to output this cycle I1122 07:04:16.413067 3798194 cpu.go:275] no items to output this cycle E1122 07:04:26.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:04:26.411949 3798194 memory.go:191] Add success. I1122 07:04:26.412298 3798194 cpu.go:282] Add success. I1122 07:04:26.441914 3798194 net.go:642] Add success. I1122 07:04:26.445744 3798194 net.go:778] primary dev: ETH0 I1122 07:04:26.445760 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:04:26.445773 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:04:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:04:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:04:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:04:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:04:36.412257 3798194 memory.go:184] no items to output this cycle I1122 07:04:36.412394 3798194 cpu.go:275] no items to output this cycle E1122 07:04:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:04:46.411972 3798194 memory.go:184] no items to output this cycle I1122 07:04:46.412125 3798194 cpu.go:275] no items to output this cycle E1122 07:04:56.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:04:56.411945 3798194 memory.go:191] Add success. W1122 07:04:56.411971 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:04:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:04:56.411986 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:04:56.412673 3798194 cpu.go:282] Add success. I1122 07:04:56.440955 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:04:56.441546 3798194 net.go:642] Add success. I1122 07:04:56.445756 3798194 net.go:778] primary dev: ETH0 I1122 07:04:56.445771 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:04:56.445783 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:04:57.450945 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:04:57.451199 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:04:57.451214 3798194 disk_worker.go:723] disk space is not compliant W1122 07:04:57.451217 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:04:57.459981 3798194 disk_worker.go:509] system disk:vda1 I1122 07:04:57.460022 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:04:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:04:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:04:59.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:04:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:04:59.471655 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:05:02.818566 3798194 disk_info.go:125] begin check local disk info of client I1122 07:05:02.822420 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:05:02.822430 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e080 0xc00030e0c0] E1122 07:05:06.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:05:06.411987 3798194 memory.go:184] no items to output this cycle I1122 07:05:06.412139 3798194 cpu.go:275] no items to output this cycle E1122 07:05:16.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:05:16.413259 3798194 memory.go:184] no items to output this cycle I1122 07:05:16.413412 3798194 cpu.go:275] no items to output this cycle E1122 07:05:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:05:26.411952 3798194 memory.go:191] Add success. I1122 07:05:26.412313 3798194 cpu.go:282] Add success. I1122 07:05:26.441914 3798194 net.go:642] Add success. I1122 07:05:26.445994 3798194 net.go:778] primary dev: ETH0 I1122 07:05:26.446011 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:05:26.446023 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:05:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:05:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:05:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:05:36.412086 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:05:36.412123 3798194 memory.go:184] no items to output this cycle I1122 07:05:36.412270 3798194 cpu.go:275] no items to output this cycle E1122 07:05:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:05:46.411983 3798194 memory.go:184] no items to output this cycle I1122 07:05:46.412134 3798194 cpu.go:275] no items to output this cycle E1122 07:05:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:05:56.411954 3798194 memory.go:191] Add success. W1122 07:05:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:05:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:05:56.412002 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:05:56.412357 3798194 cpu.go:282] Add success. I1122 07:05:56.441565 3798194 event_worker.go:152] Polling the log file for events... I1122 07:05:56.444882 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:05:56.445436 3798194 net.go:642] Add success. I1122 07:05:56.449451 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795300556, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 07:05:56.449561 3798194 net.go:778] primary dev: ETH0 I1122 07:05:56.449575 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:05:56.449588 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:05:56.480252 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 07:05:57.445477 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 07:05:57.445492 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:05:57.445755 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:05:57.445765 3798194 disk_worker.go:723] disk space is not compliant W1122 07:05:57.445768 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:05:57.447879 3798194 disk_worker.go:509] system disk:vda1 I1122 07:05:57.447919 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 07:05:58.448491 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 07:05:58.448505 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 07:05:59.449021 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 07:05:59.450097 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:05:59.450155 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:05:59.450174 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:05:59.471695 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:06:02.822534 3798194 disk_info.go:125] begin check local disk info of client I1122 07:06:02.826353 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:06:02.826361 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fc640 0xc0003fc680] E1122 07:06:06.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:06:06.413263 3798194 memory.go:184] no items to output this cycle I1122 07:06:06.413411 3798194 cpu.go:275] no items to output this cycle E1122 07:06:16.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:06:16.413254 3798194 memory.go:184] no items to output this cycle I1122 07:06:16.413398 3798194 cpu.go:275] no items to output this cycle I1122 07:06:17.654374 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:06:17.654382 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:06:26.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:06:26.413269 3798194 memory.go:191] Add success. I1122 07:06:26.417599 3798194 cpu.go:282] Add success. I1122 07:06:26.442176 3798194 net.go:642] Add success. I1122 07:06:26.443691 3798194 net.go:778] primary dev: ETH0 I1122 07:06:26.443705 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:06:26.443719 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:06:29.450038 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:06:29.450128 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:06:29.450155 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:06:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:06:36.411980 3798194 memory.go:184] no items to output this cycle I1122 07:06:36.412138 3798194 cpu.go:275] no items to output this cycle E1122 07:06:46.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:06:46.412253 3798194 memory.go:184] no items to output this cycle I1122 07:06:46.412398 3798194 cpu.go:275] no items to output this cycle E1122 07:06:56.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:06:56.413276 3798194 memory.go:191] Add success. W1122 07:06:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:06:56.413315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:06:56.413319 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:06:56.417424 3798194 cpu.go:282] Add success. I1122 07:06:56.441787 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:06:56.442355 3798194 net.go:642] Add success. I1122 07:06:56.443734 3798194 net.go:778] primary dev: ETH0 I1122 07:06:56.443748 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:06:56.443759 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:06:57.451138 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:06:57.451296 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:06:57.451348 3798194 disk_worker.go:723] disk space is not compliant W1122 07:06:57.451351 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:06:57.460077 3798194 disk_worker.go:509] system disk:vda1 I1122 07:06:57.460131 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:06:58.445864 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:06:59.449961 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:06:59.450044 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:06:59.450069 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:06:59.471650 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:07:02.826455 3798194 disk_info.go:125] begin check local disk info of client I1122 07:07:02.830298 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:07:02.830307 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000399a00 0xc000399a40] E1122 07:07:06.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:07:06.412256 3798194 memory.go:184] no items to output this cycle I1122 07:07:06.412398 3798194 cpu.go:275] no items to output this cycle E1122 07:07:16.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:07:16.412266 3798194 memory.go:184] no items to output this cycle I1122 07:07:16.412419 3798194 cpu.go:275] no items to output this cycle E1122 07:07:26.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:07:26.413265 3798194 memory.go:191] Add success. I1122 07:07:26.417333 3798194 cpu.go:282] Add success. I1122 07:07:26.441752 3798194 net.go:642] Add success. I1122 07:07:26.443175 3798194 net.go:778] primary dev: ETH0 I1122 07:07:26.443190 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:07:26.443204 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:07:29.450009 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:07:29.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:07:29.450124 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:07:36.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:07:36.412877 3798194 memory.go:184] no items to output this cycle I1122 07:07:36.413024 3798194 cpu.go:275] no items to output this cycle E1122 07:07:46.413143 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:07:46.413173 3798194 memory.go:184] no items to output this cycle I1122 07:07:46.413326 3798194 cpu.go:275] no items to output this cycle E1122 07:07:56.412211 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:07:56.412243 3798194 memory.go:191] Add success. W1122 07:07:56.412282 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:07:56.412293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:07:56.412297 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:07:56.416440 3798194 cpu.go:282] Add success. I1122 07:07:56.441825 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:07:56.442376 3798194 net.go:642] Add success. I1122 07:07:56.443743 3798194 net.go:778] primary dev: ETH0 I1122 07:07:56.443757 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:07:56.443768 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:07:57.451626 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:07:57.451835 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:07:57.451846 3798194 disk_worker.go:723] disk space is not compliant W1122 07:07:57.451849 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:07:57.463780 3798194 disk_worker.go:509] system disk:vda1 I1122 07:07:57.463824 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:07:58.445997 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:07:59.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:07:59.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:07:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:07:59.471669 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:08:02.830409 3798194 disk_info.go:125] begin check local disk info of client I1122 07:08:02.834271 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:08:02.834282 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00026da00 0xc00026da40] E1122 07:08:06.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:08:06.413259 3798194 memory.go:184] no items to output this cycle I1122 07:08:06.413403 3798194 cpu.go:275] no items to output this cycle E1122 07:08:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:08:16.413259 3798194 memory.go:184] no items to output this cycle I1122 07:08:16.413417 3798194 cpu.go:275] no items to output this cycle E1122 07:08:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:08:26.412266 3798194 memory.go:191] Add success. I1122 07:08:26.416496 3798194 cpu.go:282] Add success. I1122 07:08:26.442063 3798194 net.go:642] Add success. I1122 07:08:26.443452 3798194 net.go:778] primary dev: ETH0 I1122 07:08:26.443466 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:08:26.443480 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:08:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:08:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:08:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:08:36.411490 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:08:36.411511 3798194 memory.go:184] no items to output this cycle I1122 07:08:36.411651 3798194 cpu.go:275] no items to output this cycle E1122 07:08:46.412845 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:08:46.412867 3798194 memory.go:184] no items to output this cycle I1122 07:08:46.413010 3798194 cpu.go:275] no items to output this cycle E1122 07:08:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:08:56.411951 3798194 memory.go:191] Add success. W1122 07:08:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:08:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:08:56.412007 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:08:56.412604 3798194 cpu.go:282] Add success. I1122 07:08:56.440965 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:08:56.441554 3798194 net.go:642] Add success. I1122 07:08:56.445435 3798194 net.go:778] primary dev: ETH0 I1122 07:08:56.445450 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:08:56.445464 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:08:57.450960 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:08:57.451150 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:08:57.451204 3798194 disk_worker.go:723] disk space is not compliant W1122 07:08:57.451207 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:08:57.459972 3798194 disk_worker.go:509] system disk:vda1 I1122 07:08:57.460014 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:08:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:08:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:08:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:08:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:08:59.471655 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:09:02.834387 3798194 disk_info.go:125] begin check local disk info of client I1122 07:09:02.841714 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:09:02.841723 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030f280 0xc00030f2c0] E1122 07:09:06.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:09:06.413262 3798194 memory.go:184] no items to output this cycle I1122 07:09:06.413415 3798194 cpu.go:275] no items to output this cycle E1122 07:09:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:09:16.412258 3798194 memory.go:184] no items to output this cycle I1122 07:09:16.412411 3798194 cpu.go:275] no items to output this cycle I1122 07:09:17.655208 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:09:17.655216 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:09:26.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:09:26.413274 3798194 memory.go:191] Add success. I1122 07:09:26.417717 3798194 cpu.go:282] Add success. I1122 07:09:26.442260 3798194 net.go:642] Add success. I1122 07:09:26.443657 3798194 net.go:778] primary dev: ETH0 I1122 07:09:26.443672 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:09:26.443684 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:09:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:09:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:09:29.450504 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:09:36.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:09:36.411974 3798194 memory.go:184] no items to output this cycle I1122 07:09:36.412128 3798194 cpu.go:275] no items to output this cycle E1122 07:09:46.411942 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:09:46.411965 3798194 memory.go:184] no items to output this cycle I1122 07:09:46.412133 3798194 cpu.go:275] no items to output this cycle E1122 07:09:56.412255 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:09:56.412289 3798194 memory.go:191] Add success. W1122 07:09:56.412322 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:09:56.412333 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:09:56.412338 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:09:56.416455 3798194 cpu.go:282] Add success. I1122 07:09:56.440919 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:09:56.441521 3798194 net.go:642] Add success. I1122 07:09:56.443244 3798194 net.go:778] primary dev: ETH0 I1122 07:09:56.443258 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:09:56.443271 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:09:57.446528 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:09:57.446746 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:09:57.446815 3798194 disk_worker.go:723] disk space is not compliant W1122 07:09:57.446819 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:09:57.449689 3798194 disk_worker.go:509] system disk:vda1 I1122 07:09:57.449730 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:09:58.446387 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:09:59.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:09:59.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:09:59.450496 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:09:59.471022 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:10:02.841819 3798194 disk_info.go:125] begin check local disk info of client I1122 07:10:02.845714 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:10:02.845724 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040bc40 0xc00040bc80] E1122 07:10:06.412160 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:10:06.412186 3798194 memory.go:184] no items to output this cycle I1122 07:10:06.412529 3798194 cpu.go:275] no items to output this cycle E1122 07:10:16.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:10:16.412876 3798194 memory.go:184] no items to output this cycle I1122 07:10:16.413030 3798194 cpu.go:275] no items to output this cycle E1122 07:10:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:10:26.412268 3798194 memory.go:191] Add success. I1122 07:10:26.416538 3798194 cpu.go:282] Add success. I1122 07:10:26.442120 3798194 net.go:642] Add success. I1122 07:10:26.443670 3798194 net.go:778] primary dev: ETH0 I1122 07:10:26.443685 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:10:26.443699 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:10:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:10:29.450486 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:10:29.450518 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:10:36.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:10:36.411981 3798194 memory.go:184] no items to output this cycle I1122 07:10:36.412145 3798194 cpu.go:275] no items to output this cycle E1122 07:10:46.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:10:46.412874 3798194 memory.go:184] no items to output this cycle I1122 07:10:46.413025 3798194 cpu.go:275] no items to output this cycle E1122 07:10:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:10:56.412269 3798194 memory.go:191] Add success. W1122 07:10:56.412296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:10:56.412308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:10:56.412312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:10:56.416410 3798194 cpu.go:282] Add success. I1122 07:10:56.444720 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:10:56.445228 3798194 net.go:642] Add success. I1122 07:10:56.446745 3798194 net.go:778] primary dev: ETH0 I1122 07:10:56.446760 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:10:56.446771 3798194 net.go:659] Add eni metrics success, count: 7 E1122 07:10:57.451005 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 07:10:57.451020 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:10:57.451244 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:10:57.451255 3798194 disk_worker.go:723] disk space is not compliant W1122 07:10:57.451258 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:10:57.459920 3798194 disk_worker.go:509] system disk:vda1 I1122 07:10:57.459963 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 07:10:58.448842 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 07:10:58.448860 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 07:10:59.448634 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 07:10:59.449709 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:10:59.449763 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:10:59.449783 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:10:59.471308 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:11:02.846143 3798194 disk_info.go:125] begin check local disk info of client I1122 07:11:02.849985 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:11:02.849994 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003734c0 0xc000373500] E1122 07:11:06.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:11:06.413253 3798194 memory.go:184] no items to output this cycle I1122 07:11:06.413403 3798194 cpu.go:275] no items to output this cycle E1122 07:11:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:11:16.411980 3798194 memory.go:184] no items to output this cycle I1122 07:11:16.412130 3798194 cpu.go:275] no items to output this cycle E1122 07:11:26.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:11:26.413265 3798194 memory.go:191] Add success. I1122 07:11:26.417450 3798194 cpu.go:282] Add success. I1122 07:11:26.442256 3798194 net.go:642] Add success. I1122 07:11:26.443669 3798194 net.go:778] primary dev: ETH0 I1122 07:11:26.443685 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:11:26.443703 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:11:29.450385 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:11:29.450467 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:11:29.450493 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:11:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:11:36.411975 3798194 memory.go:184] no items to output this cycle I1122 07:11:36.412126 3798194 cpu.go:275] no items to output this cycle E1122 07:11:46.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:11:46.411969 3798194 memory.go:184] no items to output this cycle I1122 07:11:46.412131 3798194 cpu.go:275] no items to output this cycle E1122 07:11:56.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:11:56.413265 3798194 memory.go:191] Add success. W1122 07:11:56.413300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:11:56.413312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:11:56.413315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:11:56.417376 3798194 cpu.go:282] Add success. I1122 07:11:56.441774 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:11:56.442275 3798194 net.go:642] Add success. I1122 07:11:56.443874 3798194 net.go:778] primary dev: ETH0 I1122 07:11:56.443889 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:11:56.443901 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:11:57.450921 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:11:57.451143 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:11:57.451154 3798194 disk_worker.go:723] disk space is not compliant W1122 07:11:57.451157 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:11:57.459870 3798194 disk_worker.go:509] system disk:vda1 I1122 07:11:57.459913 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:11:58.445691 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:11:59.449676 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:11:59.449760 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:11:59.449785 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:11:59.471347 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:12:02.850097 3798194 disk_info.go:125] begin check local disk info of client I1122 07:12:02.853927 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:12:02.853937 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003e5900 0xc0003e5940] E1122 07:12:06.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:12:06.411972 3798194 memory.go:184] no items to output this cycle I1122 07:12:06.412123 3798194 cpu.go:275] no items to output this cycle E1122 07:12:16.413039 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:12:16.413065 3798194 memory.go:184] no items to output this cycle I1122 07:12:16.413238 3798194 cpu.go:275] no items to output this cycle I1122 07:12:17.655379 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:12:17.655388 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:12:26.411222 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:12:26.411253 3798194 memory.go:191] Add success. I1122 07:12:26.411602 3798194 cpu.go:282] Add success. I1122 07:12:26.442240 3798194 net.go:642] Add success. I1122 07:12:26.446105 3798194 net.go:778] primary dev: ETH0 I1122 07:12:26.446134 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:12:26.446149 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:12:29.450415 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:12:29.450497 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:12:29.450523 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:12:36.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:12:36.412876 3798194 memory.go:184] no items to output this cycle I1122 07:12:36.413029 3798194 cpu.go:275] no items to output this cycle E1122 07:12:46.411804 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:12:46.411827 3798194 memory.go:184] no items to output this cycle I1122 07:12:46.411970 3798194 cpu.go:275] no items to output this cycle E1122 07:12:56.413534 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:12:56.413574 3798194 memory.go:191] Add success. W1122 07:12:56.413607 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:12:56.413620 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:12:56.413624 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:12:56.418472 3798194 cpu.go:282] Add success. I1122 07:12:56.441163 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:12:56.441632 3798194 net.go:642] Add success. I1122 07:12:56.443040 3798194 net.go:778] primary dev: ETH0 I1122 07:12:56.443054 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:12:56.443067 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:12:57.449296 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:12:57.449545 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:12:57.449557 3798194 disk_worker.go:723] disk space is not compliant W1122 07:12:57.449561 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:12:57.451416 3798194 disk_worker.go:509] system disk:vda1 I1122 07:12:57.451457 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:12:58.446393 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:12:59.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:12:59.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:12:59.450515 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:12:59.471061 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:13:02.855908 3798194 disk_info.go:125] begin check local disk info of client I1122 07:13:02.859863 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:13:02.859874 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030eec0 0xc00030ef00] E1122 07:13:06.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:13:06.412262 3798194 memory.go:184] no items to output this cycle I1122 07:13:06.412405 3798194 cpu.go:275] no items to output this cycle E1122 07:13:16.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:13:16.413252 3798194 memory.go:184] no items to output this cycle I1122 07:13:16.413397 3798194 cpu.go:275] no items to output this cycle E1122 07:13:26.411907 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:13:26.411952 3798194 memory.go:191] Add success. I1122 07:13:26.412301 3798194 cpu.go:282] Add success. I1122 07:13:26.442056 3798194 net.go:642] Add success. I1122 07:13:26.445839 3798194 net.go:778] primary dev: ETH0 I1122 07:13:26.445854 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:13:26.445866 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:13:29.450005 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:13:29.450083 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:13:29.450119 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:13:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:13:36.412252 3798194 memory.go:184] no items to output this cycle I1122 07:13:36.412408 3798194 cpu.go:275] no items to output this cycle E1122 07:13:46.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:13:46.411974 3798194 memory.go:184] no items to output this cycle I1122 07:13:46.412129 3798194 cpu.go:275] no items to output this cycle E1122 07:13:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:13:56.411949 3798194 memory.go:191] Add success. W1122 07:13:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:13:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:13:56.411993 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:13:56.412339 3798194 cpu.go:282] Add success. I1122 07:13:56.441768 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:13:56.442276 3798194 net.go:642] Add success. I1122 07:13:56.445980 3798194 net.go:778] primary dev: ETH0 I1122 07:13:56.445996 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:13:56.446009 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:13:57.450936 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:13:57.451169 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:13:57.451181 3798194 disk_worker.go:723] disk space is not compliant W1122 07:13:57.451184 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:13:57.460630 3798194 disk_worker.go:509] system disk:vda1 I1122 07:13:57.460664 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:13:58.446199 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:13:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:13:59.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:13:59.450146 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:13:59.471689 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:14:02.861148 3798194 disk_info.go:125] begin check local disk info of client I1122 07:14:02.865004 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:14:02.865013 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000419840 0xc000419880] E1122 07:14:06.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:14:06.413264 3798194 memory.go:184] no items to output this cycle I1122 07:14:06.413412 3798194 cpu.go:275] no items to output this cycle E1122 07:14:16.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:14:16.411975 3798194 memory.go:184] no items to output this cycle I1122 07:14:16.412137 3798194 cpu.go:275] no items to output this cycle E1122 07:14:26.411891 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:14:26.411924 3798194 memory.go:191] Add success. I1122 07:14:26.412286 3798194 cpu.go:282] Add success. I1122 07:14:26.441922 3798194 net.go:642] Add success. I1122 07:14:26.445854 3798194 net.go:778] primary dev: ETH0 I1122 07:14:26.445869 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:14:26.445884 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:14:29.450445 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:14:29.450526 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:14:29.450552 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:14:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:14:36.412258 3798194 memory.go:184] no items to output this cycle I1122 07:14:36.412398 3798194 cpu.go:275] no items to output this cycle E1122 07:14:46.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:14:46.412269 3798194 memory.go:184] no items to output this cycle I1122 07:14:46.412423 3798194 cpu.go:275] no items to output this cycle E1122 07:14:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:14:56.411952 3798194 memory.go:191] Add success. W1122 07:14:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:14:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:14:56.412000 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:14:56.412330 3798194 cpu.go:282] Add success. I1122 07:14:56.441764 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:14:56.442330 3798194 net.go:642] Add success. I1122 07:14:56.446551 3798194 net.go:778] primary dev: ETH0 I1122 07:14:56.446567 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:14:56.446589 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:14:57.450627 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:14:57.450791 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:14:57.450846 3798194 disk_worker.go:723] disk space is not compliant W1122 07:14:57.450849 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:14:57.458844 3798194 disk_worker.go:509] system disk:vda1 I1122 07:14:57.458890 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:14:58.446010 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:14:59.450414 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:14:59.450492 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:14:59.450520 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:14:59.471012 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:15:02.872139 3798194 disk_info.go:125] begin check local disk info of client I1122 07:15:02.888521 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:15:02.888531 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a24c0 0xc0003a2500] E1122 07:15:06.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:15:06.412261 3798194 memory.go:184] no items to output this cycle I1122 07:15:06.412406 3798194 cpu.go:275] no items to output this cycle E1122 07:15:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:15:16.411979 3798194 memory.go:184] no items to output this cycle I1122 07:15:16.412136 3798194 cpu.go:275] no items to output this cycle I1122 07:15:17.655559 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:15:17.655567 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:15:26.412573 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:15:26.412611 3798194 memory.go:191] Add success. I1122 07:15:26.417462 3798194 cpu.go:282] Add success. I1122 07:15:26.442006 3798194 net.go:642] Add success. I1122 07:15:26.443408 3798194 net.go:778] primary dev: ETH0 I1122 07:15:26.443423 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:15:26.443435 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:15:29.450239 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:15:29.450319 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:15:29.450352 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:15:36.413312 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:15:36.413332 3798194 memory.go:184] no items to output this cycle I1122 07:15:36.413475 3798194 cpu.go:275] no items to output this cycle E1122 07:15:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:15:46.411975 3798194 memory.go:184] no items to output this cycle I1122 07:15:46.412132 3798194 cpu.go:275] no items to output this cycle E1122 07:15:56.413108 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:15:56.413152 3798194 memory.go:191] Add success. W1122 07:15:56.413183 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:15:56.413194 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:15:56.413198 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:15:56.417172 3798194 cpu.go:282] Add success. I1122 07:15:56.444230 3798194 event_worker.go:152] Polling the log file for events... I1122 07:15:56.444448 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:15:56.444952 3798194 net.go:642] Add success. I1122 07:15:56.446329 3798194 net.go:778] primary dev: ETH0 I1122 07:15:56.446343 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:15:56.446355 3798194 net.go:659] Add eni metrics success, count: 7 E1122 07:15:57.450989 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 07:15:57.451003 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:15:57.451241 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:15:57.451251 3798194 disk_worker.go:723] disk space is not compliant W1122 07:15:57.451254 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:15:57.460130 3798194 disk_worker.go:509] system disk:vda1 I1122 07:15:57.460174 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 07:15:58.448533 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 07:15:58.448547 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 07:15:59.448613 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 07:15:59.449689 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:15:59.449739 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:15:59.449757 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:15:59.471183 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:16:02.888639 3798194 disk_info.go:125] begin check local disk info of client I1122 07:16:02.892507 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:16:02.892516 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000309600 0xc000309640] E1122 07:16:06.411971 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:16:06.411997 3798194 memory.go:184] no items to output this cycle I1122 07:16:06.412165 3798194 cpu.go:275] no items to output this cycle E1122 07:16:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:16:16.411974 3798194 memory.go:184] no items to output this cycle I1122 07:16:16.412130 3798194 cpu.go:275] no items to output this cycle E1122 07:16:26.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:16:26.411965 3798194 memory.go:191] Add success. I1122 07:16:26.412323 3798194 cpu.go:282] Add success. I1122 07:16:26.441971 3798194 net.go:642] Add success. I1122 07:16:26.445807 3798194 net.go:778] primary dev: ETH0 I1122 07:16:26.445822 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:16:26.445840 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:16:29.450008 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:16:29.450090 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:16:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:16:36.412759 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:16:36.412782 3798194 memory.go:184] no items to output this cycle I1122 07:16:36.412937 3798194 cpu.go:275] no items to output this cycle E1122 07:16:46.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:16:46.413271 3798194 memory.go:184] no items to output this cycle I1122 07:16:46.413417 3798194 cpu.go:275] no items to output this cycle E1122 07:16:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:16:56.411958 3798194 memory.go:191] Add success. W1122 07:16:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:16:56.412004 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:16:56.412008 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:16:56.412373 3798194 cpu.go:282] Add success. I1122 07:16:56.441832 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:16:56.442365 3798194 net.go:642] Add success. I1122 07:16:56.446592 3798194 net.go:778] primary dev: ETH0 I1122 07:16:56.446608 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:16:56.446620 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:16:57.450939 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:16:57.451165 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:16:57.451176 3798194 disk_worker.go:723] disk space is not compliant W1122 07:16:57.451179 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:16:57.459836 3798194 disk_worker.go:509] system disk:vda1 I1122 07:16:57.459880 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:16:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:16:59.450033 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:16:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:16:59.450145 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:16:59.471713 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:17:02.892622 3798194 disk_info.go:125] begin check local disk info of client I1122 07:17:02.896524 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:17:02.896533 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000368fc0 0xc000369000] E1122 07:17:06.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:17:06.413263 3798194 memory.go:184] no items to output this cycle I1122 07:17:06.413405 3798194 cpu.go:275] no items to output this cycle E1122 07:17:16.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:17:16.412874 3798194 memory.go:184] no items to output this cycle I1122 07:17:16.413034 3798194 cpu.go:275] no items to output this cycle E1122 07:17:26.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:17:26.412883 3798194 memory.go:191] Add success. I1122 07:17:26.417096 3798194 cpu.go:282] Add success. I1122 07:17:26.441632 3798194 net.go:642] Add success. I1122 07:17:26.443177 3798194 net.go:778] primary dev: ETH0 I1122 07:17:26.443191 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:17:26.443204 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:17:29.450386 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:17:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:17:29.450495 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:17:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:17:36.411978 3798194 memory.go:184] no items to output this cycle I1122 07:17:36.412130 3798194 cpu.go:275] no items to output this cycle E1122 07:17:46.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:17:46.411990 3798194 memory.go:184] no items to output this cycle I1122 07:17:46.412150 3798194 cpu.go:275] no items to output this cycle E1122 07:17:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:17:56.411951 3798194 memory.go:191] Add success. W1122 07:17:56.411980 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:17:56.411991 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:17:56.411995 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:17:56.412350 3798194 cpu.go:282] Add success. I1122 07:17:56.441801 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:17:56.442312 3798194 net.go:642] Add success. I1122 07:17:56.446149 3798194 net.go:778] primary dev: ETH0 I1122 07:17:56.446165 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:17:56.446339 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:17:57.450922 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:17:57.451079 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:17:57.451159 3798194 disk_worker.go:723] disk space is not compliant W1122 07:17:57.451162 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:17:57.459944 3798194 disk_worker.go:509] system disk:vda1 I1122 07:17:57.459987 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:17:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:17:59.450413 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:17:59.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:17:59.450508 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:17:59.471076 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:18:02.898145 3798194 disk_info.go:125] begin check local disk info of client I1122 07:18:02.901996 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:18:02.902005 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000070e00 0xc000070e40] E1122 07:18:06.412418 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:18:06.412441 3798194 memory.go:184] no items to output this cycle I1122 07:18:06.412586 3798194 cpu.go:275] no items to output this cycle E1122 07:18:16.412189 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:18:16.412210 3798194 memory.go:184] no items to output this cycle I1122 07:18:16.412357 3798194 cpu.go:275] no items to output this cycle I1122 07:18:17.655727 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:18:17.655736 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:18:26.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:18:26.412897 3798194 memory.go:191] Add success. I1122 07:18:26.417411 3798194 cpu.go:282] Add success. I1122 07:18:26.441960 3798194 net.go:642] Add success. I1122 07:18:26.443405 3798194 net.go:778] primary dev: ETH0 I1122 07:18:26.443419 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:18:26.443432 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:18:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:18:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:18:29.450498 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:18:36.412223 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:18:36.412256 3798194 memory.go:184] no items to output this cycle I1122 07:18:36.412396 3798194 cpu.go:275] no items to output this cycle E1122 07:18:46.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:18:46.411971 3798194 memory.go:184] no items to output this cycle I1122 07:18:46.412136 3798194 cpu.go:275] no items to output this cycle E1122 07:18:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:18:56.411949 3798194 memory.go:191] Add success. W1122 07:18:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:18:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:18:56.411991 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:18:56.412339 3798194 cpu.go:282] Add success. I1122 07:18:56.440956 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:18:56.441564 3798194 net.go:642] Add success. I1122 07:18:56.445649 3798194 net.go:778] primary dev: ETH0 I1122 07:18:56.445667 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:18:56.445679 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:18:57.445599 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:18:57.445864 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:18:57.445876 3798194 disk_worker.go:723] disk space is not compliant W1122 07:18:57.445879 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:18:57.450845 3798194 disk_worker.go:509] system disk:vda1 I1122 07:18:57.450882 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:18:58.445982 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:18:59.450491 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:18:59.450576 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:18:59.450602 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:18:59.471132 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:19:02.903135 3798194 disk_info.go:125] begin check local disk info of client I1122 07:19:02.907037 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:19:02.907047 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040b580 0xc00040b5c0] E1122 07:19:06.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:19:06.412879 3798194 memory.go:184] no items to output this cycle I1122 07:19:06.413021 3798194 cpu.go:275] no items to output this cycle E1122 07:19:16.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:19:16.411986 3798194 memory.go:184] no items to output this cycle I1122 07:19:16.412142 3798194 cpu.go:275] no items to output this cycle E1122 07:19:26.412975 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:19:26.413075 3798194 memory.go:191] Add success. I1122 07:19:26.417821 3798194 cpu.go:282] Add success. I1122 07:19:26.441660 3798194 net.go:642] Add success. I1122 07:19:26.443250 3798194 net.go:778] primary dev: ETH0 I1122 07:19:26.443266 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:19:26.443288 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:19:29.450385 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:19:29.450470 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:19:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:19:36.412814 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:19:36.412835 3798194 memory.go:184] no items to output this cycle I1122 07:19:36.412979 3798194 cpu.go:275] no items to output this cycle E1122 07:19:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:19:46.411974 3798194 memory.go:184] no items to output this cycle I1122 07:19:46.412135 3798194 cpu.go:275] no items to output this cycle E1122 07:19:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:19:56.411952 3798194 memory.go:191] Add success. W1122 07:19:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:19:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:19:56.411993 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:19:56.412350 3798194 cpu.go:282] Add success. I1122 07:19:56.441797 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:19:56.442313 3798194 net.go:642] Add success. I1122 07:19:56.445909 3798194 net.go:778] primary dev: ETH0 I1122 07:19:56.445924 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:19:56.445936 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:19:57.450594 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:19:57.450800 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:19:57.450812 3798194 disk_worker.go:723] disk space is not compliant W1122 07:19:57.450816 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:19:57.459155 3798194 disk_worker.go:509] system disk:vda1 I1122 07:19:57.459198 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:19:58.445920 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:19:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:19:59.450121 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:19:59.450154 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:19:59.471716 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:20:02.908152 3798194 disk_info.go:125] begin check local disk info of client I1122 07:20:02.913991 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:20:02.914000 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000322c40 0xc000322c80] E1122 07:20:06.412653 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:20:06.412677 3798194 memory.go:184] no items to output this cycle I1122 07:20:06.412819 3798194 cpu.go:275] no items to output this cycle E1122 07:20:16.413405 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:20:16.413427 3798194 memory.go:184] no items to output this cycle I1122 07:20:16.413571 3798194 cpu.go:275] no items to output this cycle E1122 07:20:26.412224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:20:26.412261 3798194 memory.go:191] Add success. I1122 07:20:26.416361 3798194 cpu.go:282] Add success. I1122 07:20:26.441960 3798194 net.go:642] Add success. I1122 07:20:26.443503 3798194 net.go:778] primary dev: ETH0 I1122 07:20:26.443517 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:20:26.443534 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:20:29.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:20:29.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:20:29.450498 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:20:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:20:36.413260 3798194 memory.go:184] no items to output this cycle I1122 07:20:36.413404 3798194 cpu.go:275] no items to output this cycle E1122 07:20:46.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:20:46.413249 3798194 memory.go:184] no items to output this cycle I1122 07:20:46.413399 3798194 cpu.go:275] no items to output this cycle E1122 07:20:56.412678 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:20:56.412711 3798194 memory.go:191] Add success. W1122 07:20:56.412743 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:20:56.412754 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:20:56.412758 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:20:56.416874 3798194 cpu.go:282] Add success. I1122 07:20:56.443903 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:20:56.444513 3798194 net.go:642] Add success. I1122 07:20:56.446025 3798194 net.go:778] primary dev: ETH0 I1122 07:20:56.446040 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:20:56.446052 3798194 net.go:659] Add eni metrics success, count: 7 E1122 07:20:57.450433 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 07:20:57.450448 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:20:57.450641 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:20:57.450651 3798194 disk_worker.go:723] disk space is not compliant W1122 07:20:57.450654 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:20:57.459222 3798194 disk_worker.go:509] system disk:vda1 I1122 07:20:57.459265 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 07:20:58.448064 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 07:20:58.448077 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 07:20:59.448735 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 07:20:59.449820 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:20:59.449873 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:20:59.449896 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:20:59.471432 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:21:02.915148 3798194 disk_info.go:125] begin check local disk info of client I1122 07:21:02.919019 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:21:02.919029 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e600 0xc00030e640] E1122 07:21:06.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:21:06.413256 3798194 memory.go:184] no items to output this cycle I1122 07:21:06.413412 3798194 cpu.go:275] no items to output this cycle E1122 07:21:16.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:21:16.411984 3798194 memory.go:184] no items to output this cycle I1122 07:21:16.412143 3798194 cpu.go:275] no items to output this cycle I1122 07:21:17.656203 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:21:17.656212 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:21:26.412612 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:21:26.412645 3798194 memory.go:191] Add success. I1122 07:21:26.416939 3798194 cpu.go:282] Add success. I1122 07:21:26.441542 3798194 net.go:642] Add success. I1122 07:21:26.443022 3798194 net.go:778] primary dev: ETH0 I1122 07:21:26.443036 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:21:26.443048 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:21:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:21:29.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:21:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:21:36.411984 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:21:36.412008 3798194 memory.go:184] no items to output this cycle I1122 07:21:36.412181 3798194 cpu.go:275] no items to output this cycle E1122 07:21:46.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:21:46.411989 3798194 memory.go:184] no items to output this cycle I1122 07:21:46.412149 3798194 cpu.go:275] no items to output this cycle E1122 07:21:56.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:21:56.412888 3798194 memory.go:191] Add success. W1122 07:21:56.412924 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:21:56.412936 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:21:56.412939 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:21:56.417398 3798194 cpu.go:282] Add success. I1122 07:21:56.441778 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:21:56.442287 3798194 net.go:642] Add success. I1122 07:21:56.443662 3798194 net.go:778] primary dev: ETH0 I1122 07:21:56.443675 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:21:56.443687 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:21:57.451156 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:21:57.451385 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:21:57.451396 3798194 disk_worker.go:723] disk space is not compliant W1122 07:21:57.451400 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:21:57.459963 3798194 disk_worker.go:509] system disk:vda1 I1122 07:21:57.460004 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:21:58.446530 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:21:59.450626 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:21:59.450720 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:21:59.450747 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:21:59.471192 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:22:02.920137 3798194 disk_info.go:125] begin check local disk info of client I1122 07:22:02.924033 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:22:02.924042 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000457340 0xc000457380] E1122 07:22:06.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:22:06.413263 3798194 memory.go:184] no items to output this cycle I1122 07:22:06.413408 3798194 cpu.go:275] no items to output this cycle E1122 07:22:16.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:22:16.413268 3798194 memory.go:184] no items to output this cycle I1122 07:22:16.413413 3798194 cpu.go:275] no items to output this cycle E1122 07:22:26.412259 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:22:26.412293 3798194 memory.go:191] Add success. I1122 07:22:26.416487 3798194 cpu.go:282] Add success. I1122 07:22:26.442076 3798194 net.go:642] Add success. I1122 07:22:26.443595 3798194 net.go:778] primary dev: ETH0 I1122 07:22:26.443610 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:22:26.443623 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:22:29.450422 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:22:29.450491 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:22:29.450527 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:22:36.412871 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:22:36.412896 3798194 memory.go:184] no items to output this cycle I1122 07:22:36.413072 3798194 cpu.go:275] no items to output this cycle E1122 07:22:46.411970 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:22:46.411995 3798194 memory.go:184] no items to output this cycle I1122 07:22:46.412171 3798194 cpu.go:275] no items to output this cycle E1122 07:22:56.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:22:56.412274 3798194 memory.go:191] Add success. W1122 07:22:56.412304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:22:56.412315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:22:56.412320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:22:56.416551 3798194 cpu.go:282] Add success. I1122 07:22:56.441956 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:22:56.442555 3798194 net.go:642] Add success. I1122 07:22:56.444065 3798194 net.go:778] primary dev: ETH0 I1122 07:22:56.444079 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:22:56.444097 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:22:57.450749 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:22:57.450952 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:22:57.450963 3798194 disk_worker.go:723] disk space is not compliant W1122 07:22:57.450966 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:22:57.459369 3798194 disk_worker.go:509] system disk:vda1 I1122 07:22:57.459407 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:22:58.446383 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:22:59.450479 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:22:59.450561 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:22:59.450590 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:22:59.471116 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:23:02.924159 3798194 disk_info.go:125] begin check local disk info of client I1122 07:23:02.929321 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:23:02.929331 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000272340 0xc000272380] E1122 07:23:06.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:23:06.413268 3798194 memory.go:184] no items to output this cycle I1122 07:23:06.413422 3798194 cpu.go:275] no items to output this cycle E1122 07:23:16.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:23:16.413249 3798194 memory.go:184] no items to output this cycle I1122 07:23:16.413400 3798194 cpu.go:275] no items to output this cycle E1122 07:23:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:23:26.411953 3798194 memory.go:191] Add success. I1122 07:23:26.412329 3798194 cpu.go:282] Add success. I1122 07:23:26.441971 3798194 net.go:642] Add success. I1122 07:23:26.446279 3798194 net.go:778] primary dev: ETH0 I1122 07:23:26.446294 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:23:26.446306 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:23:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:23:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:23:29.450134 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:23:36.412097 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:23:36.412138 3798194 memory.go:184] no items to output this cycle I1122 07:23:36.412282 3798194 cpu.go:275] no items to output this cycle E1122 07:23:46.413090 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:23:46.413125 3798194 memory.go:184] no items to output this cycle I1122 07:23:46.413468 3798194 cpu.go:275] no items to output this cycle E1122 07:23:56.411972 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:23:56.412005 3798194 memory.go:191] Add success. W1122 07:23:56.412040 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:23:56.412053 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:23:56.412057 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:23:56.416255 3798194 cpu.go:282] Add success. I1122 07:23:56.441618 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:23:56.442094 3798194 net.go:642] Add success. I1122 07:23:56.443837 3798194 net.go:778] primary dev: ETH0 I1122 07:23:56.443851 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:23:56.443865 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:23:57.451596 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:23:57.451803 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:23:57.451814 3798194 disk_worker.go:723] disk space is not compliant W1122 07:23:57.451817 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:23:57.461167 3798194 disk_worker.go:509] system disk:vda1 I1122 07:23:57.461208 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:23:58.446677 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:23:59.449669 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:23:59.449758 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:23:59.449784 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:23:59.471354 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:24:02.931148 3798194 disk_info.go:125] begin check local disk info of client I1122 07:24:02.935063 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:24:02.935073 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a64c0 0xc0003a6500] E1122 07:24:06.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:24:06.412873 3798194 memory.go:184] no items to output this cycle I1122 07:24:06.413018 3798194 cpu.go:275] no items to output this cycle E1122 07:24:16.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:24:16.412256 3798194 memory.go:184] no items to output this cycle I1122 07:24:16.412402 3798194 cpu.go:275] no items to output this cycle I1122 07:24:17.657214 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:24:17.657222 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:24:26.411302 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:24:26.411333 3798194 memory.go:191] Add success. I1122 07:24:26.411673 3798194 cpu.go:282] Add success. I1122 07:24:26.441268 3798194 net.go:642] Add success. I1122 07:24:26.445390 3798194 net.go:778] primary dev: ETH0 I1122 07:24:26.445406 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:24:26.445419 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:24:29.450016 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:24:29.450125 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:24:29.450159 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:24:36.413167 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:24:36.413193 3798194 memory.go:184] no items to output this cycle I1122 07:24:36.413336 3798194 cpu.go:275] no items to output this cycle E1122 07:24:46.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:24:46.413272 3798194 memory.go:184] no items to output this cycle I1122 07:24:46.413423 3798194 cpu.go:275] no items to output this cycle E1122 07:24:56.411919 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:24:56.411950 3798194 memory.go:191] Add success. W1122 07:24:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:24:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:24:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:24:56.412336 3798194 cpu.go:282] Add success. I1122 07:24:56.441795 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:24:56.442351 3798194 net.go:642] Add success. I1122 07:24:56.446008 3798194 net.go:778] primary dev: ETH0 I1122 07:24:56.446024 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:24:56.446037 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:24:57.450558 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:24:57.450733 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:24:57.450789 3798194 disk_worker.go:723] disk space is not compliant W1122 07:24:57.450792 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:24:57.459974 3798194 disk_worker.go:509] system disk:vda1 I1122 07:24:57.460016 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:24:58.446364 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:24:59.450462 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:24:59.450557 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:24:59.450583 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:24:59.471124 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:25:02.937145 3798194 disk_info.go:125] begin check local disk info of client I1122 07:25:02.940952 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:25:02.940962 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000493500 0xc000493540] E1122 07:25:06.412753 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:25:06.412777 3798194 memory.go:184] no items to output this cycle I1122 07:25:06.412928 3798194 cpu.go:275] no items to output this cycle E1122 07:25:16.412226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:25:16.412249 3798194 memory.go:184] no items to output this cycle I1122 07:25:16.412392 3798194 cpu.go:275] no items to output this cycle E1122 07:25:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:25:26.411961 3798194 memory.go:191] Add success. I1122 07:25:26.412317 3798194 cpu.go:282] Add success. I1122 07:25:26.441913 3798194 net.go:642] Add success. I1122 07:25:26.446392 3798194 net.go:778] primary dev: ETH0 I1122 07:25:26.446408 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:25:26.446421 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:25:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:25:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:25:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:25:36.412759 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:25:36.412781 3798194 memory.go:184] no items to output this cycle I1122 07:25:36.412932 3798194 cpu.go:275] no items to output this cycle E1122 07:25:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:25:46.411978 3798194 memory.go:184] no items to output this cycle I1122 07:25:46.412146 3798194 cpu.go:275] no items to output this cycle E1122 07:25:56.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:25:56.413269 3798194 memory.go:191] Add success. W1122 07:25:56.413302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:25:56.413314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:25:56.413317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:25:56.417467 3798194 cpu.go:282] Add success. I1122 07:25:56.441604 3798194 event_worker.go:152] Polling the log file for events... I1122 07:25:56.444727 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:25:56.445296 3798194 net.go:642] Add success. I1122 07:25:56.446817 3798194 net.go:778] primary dev: ETH0 I1122 07:25:56.446831 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:25:56.446843 3798194 net.go:659] Add eni metrics success, count: 7 E1122 07:25:57.450292 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 07:25:57.450306 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:25:57.450464 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:25:57.450518 3798194 disk_worker.go:723] disk space is not compliant W1122 07:25:57.450521 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:25:57.458541 3798194 disk_worker.go:509] system disk:vda1 I1122 07:25:57.458583 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 07:25:58.448243 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 07:25:58.448258 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 07:25:59.448634 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 07:25:59.449709 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:25:59.449758 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:25:59.449778 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:25:59.471311 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:26:02.941056 3798194 disk_info.go:125] begin check local disk info of client I1122 07:26:02.944920 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:26:02.944929 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003a8cc0 0xc0003a8d00] E1122 07:26:06.411773 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:26:06.411795 3798194 memory.go:184] no items to output this cycle I1122 07:26:06.411938 3798194 cpu.go:275] no items to output this cycle E1122 07:26:16.412998 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:26:16.413023 3798194 memory.go:184] no items to output this cycle I1122 07:26:16.413187 3798194 cpu.go:275] no items to output this cycle E1122 07:26:26.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:26:26.412277 3798194 memory.go:191] Add success. I1122 07:26:26.416457 3798194 cpu.go:282] Add success. I1122 07:26:26.442014 3798194 net.go:642] Add success. I1122 07:26:26.443567 3798194 net.go:778] primary dev: ETH0 I1122 07:26:26.443581 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:26:26.443594 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:26:29.450402 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:26:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:26:29.450512 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:26:36.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:26:36.411983 3798194 memory.go:184] no items to output this cycle I1122 07:26:36.412144 3798194 cpu.go:275] no items to output this cycle E1122 07:26:46.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:26:46.413262 3798194 memory.go:184] no items to output this cycle I1122 07:26:46.413408 3798194 cpu.go:275] no items to output this cycle E1122 07:26:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:26:56.411943 3798194 memory.go:191] Add success. W1122 07:26:56.411974 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:26:56.411986 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:26:56.411990 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:26:56.412317 3798194 cpu.go:282] Add success. I1122 07:26:56.441771 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:26:56.442267 3798194 net.go:642] Add success. I1122 07:26:56.446465 3798194 net.go:778] primary dev: ETH0 I1122 07:26:56.446480 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:26:56.446492 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:26:57.450582 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:26:57.450752 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:26:57.450807 3798194 disk_worker.go:723] disk space is not compliant W1122 07:26:57.450811 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:26:57.459189 3798194 disk_worker.go:509] system disk:vda1 I1122 07:26:57.459227 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:26:58.446038 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:26:59.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:26:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:26:59.450502 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:26:59.470997 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:27:02.947154 3798194 disk_info.go:125] begin check local disk info of client I1122 07:27:02.951025 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:27:02.951035 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000313480 0xc0003134c0] E1122 07:27:06.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:27:06.413254 3798194 memory.go:184] no items to output this cycle I1122 07:27:06.413400 3798194 cpu.go:275] no items to output this cycle E1122 07:27:16.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:27:16.411976 3798194 memory.go:184] no items to output this cycle I1122 07:27:16.412135 3798194 cpu.go:275] no items to output this cycle I1122 07:27:17.657386 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:27:17.657398 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:27:26.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:27:26.412260 3798194 memory.go:191] Add success. I1122 07:27:26.417229 3798194 cpu.go:282] Add success. I1122 07:27:26.441728 3798194 net.go:642] Add success. I1122 07:27:26.443241 3798194 net.go:778] primary dev: ETH0 I1122 07:27:26.443256 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:27:26.443268 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:27:29.450194 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:27:29.450277 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:27:29.450305 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:27:36.413253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:27:36.413277 3798194 memory.go:184] no items to output this cycle I1122 07:27:36.413429 3798194 cpu.go:275] no items to output this cycle E1122 07:27:46.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:27:46.412087 3798194 memory.go:184] no items to output this cycle I1122 07:27:46.412440 3798194 cpu.go:275] no items to output this cycle E1122 07:27:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:27:56.411963 3798194 memory.go:191] Add success. W1122 07:27:56.411997 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:27:56.412009 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:27:56.412012 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:27:56.412386 3798194 cpu.go:282] Add success. I1122 07:27:56.441838 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:27:56.442437 3798194 net.go:642] Add success. I1122 07:27:56.446352 3798194 net.go:778] primary dev: ETH0 I1122 07:27:56.446367 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:27:56.446380 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:27:57.450696 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:27:57.450916 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:27:57.450928 3798194 disk_worker.go:723] disk space is not compliant W1122 07:27:57.450930 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:27:57.459284 3798194 disk_worker.go:509] system disk:vda1 I1122 07:27:57.459326 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:27:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:27:59.450107 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:27:59.450198 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:27:59.450225 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:27:59.470728 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:28:02.952149 3798194 disk_info.go:125] begin check local disk info of client I1122 07:28:02.956096 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:28:02.956105 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000314e40 0xc000314e80] E1122 07:28:06.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:28:06.413254 3798194 memory.go:184] no items to output this cycle I1122 07:28:06.413406 3798194 cpu.go:275] no items to output this cycle E1122 07:28:16.411245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:28:16.411270 3798194 memory.go:184] no items to output this cycle I1122 07:28:16.411418 3798194 cpu.go:275] no items to output this cycle E1122 07:28:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:28:26.411952 3798194 memory.go:191] Add success. I1122 07:28:26.412314 3798194 cpu.go:282] Add success. I1122 07:28:26.441904 3798194 net.go:642] Add success. I1122 07:28:26.445704 3798194 net.go:778] primary dev: ETH0 I1122 07:28:26.445720 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:28:26.445734 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:28:29.450006 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:28:29.450080 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:28:29.450105 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:28:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:28:36.412254 3798194 memory.go:184] no items to output this cycle I1122 07:28:36.412397 3798194 cpu.go:275] no items to output this cycle E1122 07:28:46.412813 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:28:46.412835 3798194 memory.go:184] no items to output this cycle I1122 07:28:46.412993 3798194 cpu.go:275] no items to output this cycle E1122 07:28:56.413225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:28:56.413259 3798194 memory.go:191] Add success. W1122 07:28:56.413284 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:28:56.413295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:28:56.413299 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:28:56.417541 3798194 cpu.go:282] Add success. I1122 07:28:56.441917 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:28:56.442487 3798194 net.go:642] Add success. I1122 07:28:56.443983 3798194 net.go:778] primary dev: ETH0 I1122 07:28:56.443997 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:28:56.444010 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:28:57.450979 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:28:57.451156 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:28:57.451209 3798194 disk_worker.go:723] disk space is not compliant W1122 07:28:57.451212 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:28:57.460831 3798194 disk_worker.go:509] system disk:vda1 I1122 07:28:57.460871 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:28:58.446367 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:28:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:28:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:28:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:28:59.471651 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:29:02.956232 3798194 disk_info.go:125] begin check local disk info of client I1122 07:29:02.960070 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:29:02.960078 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000457580 0xc0004575c0] E1122 07:29:06.411808 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:29:06.411837 3798194 memory.go:184] no items to output this cycle I1122 07:29:06.411985 3798194 cpu.go:275] no items to output this cycle E1122 07:29:16.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:29:16.411971 3798194 memory.go:184] no items to output this cycle I1122 07:29:16.412131 3798194 cpu.go:275] no items to output this cycle E1122 07:29:26.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:29:26.411948 3798194 memory.go:191] Add success. I1122 07:29:26.412298 3798194 cpu.go:282] Add success. I1122 07:29:26.442034 3798194 net.go:642] Add success. I1122 07:29:26.446188 3798194 net.go:778] primary dev: ETH0 I1122 07:29:26.446203 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:29:26.446217 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:29:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:29:29.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:29:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:29:36.411945 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:29:36.411969 3798194 memory.go:184] no items to output this cycle I1122 07:29:36.412134 3798194 cpu.go:275] no items to output this cycle E1122 07:29:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:29:46.411970 3798194 memory.go:184] no items to output this cycle I1122 07:29:46.412128 3798194 cpu.go:275] no items to output this cycle E1122 07:29:56.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:29:56.413261 3798194 memory.go:191] Add success. W1122 07:29:56.413287 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:29:56.413298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:29:56.413302 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:29:56.417323 3798194 cpu.go:282] Add success. I1122 07:29:56.441830 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:29:56.442401 3798194 net.go:642] Add success. I1122 07:29:56.443870 3798194 net.go:778] primary dev: ETH0 I1122 07:29:56.443885 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:29:56.443898 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:29:57.450945 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:29:57.451167 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:29:57.451177 3798194 disk_worker.go:723] disk space is not compliant W1122 07:29:57.451181 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:29:57.459952 3798194 disk_worker.go:509] system disk:vda1 I1122 07:29:57.459996 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:29:58.446370 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:29:59.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:29:59.450463 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:29:59.450488 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:29:59.471023 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:30:02.961137 3798194 disk_info.go:125] begin check local disk info of client I1122 07:30:02.964989 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:30:02.964998 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c8c40 0xc0003c8c80] E1122 07:30:06.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:30:06.412876 3798194 memory.go:184] no items to output this cycle I1122 07:30:06.413018 3798194 cpu.go:275] no items to output this cycle E1122 07:30:16.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:30:16.413273 3798194 memory.go:184] no items to output this cycle I1122 07:30:16.413415 3798194 cpu.go:275] no items to output this cycle I1122 07:30:17.657569 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:30:17.657578 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:30:26.412973 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:30:26.413012 3798194 memory.go:191] Add success. I1122 07:30:26.417246 3798194 cpu.go:282] Add success. I1122 07:30:26.441938 3798194 net.go:642] Add success. I1122 07:30:26.443525 3798194 net.go:778] primary dev: ETH0 I1122 07:30:26.443540 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:30:26.443553 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:30:29.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:30:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:30:29.450514 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:30:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:30:36.411978 3798194 memory.go:184] no items to output this cycle I1122 07:30:36.412133 3798194 cpu.go:275] no items to output this cycle E1122 07:30:46.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:30:46.411971 3798194 memory.go:184] no items to output this cycle I1122 07:30:46.412133 3798194 cpu.go:275] no items to output this cycle E1122 07:30:56.411906 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:30:56.411938 3798194 memory.go:191] Add success. W1122 07:30:56.411962 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:30:56.411974 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:30:56.411978 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:30:56.412320 3798194 cpu.go:282] Add success. I1122 07:30:56.444672 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:30:56.445172 3798194 net.go:642] Add success. I1122 07:30:56.449366 3798194 net.go:778] primary dev: ETH0 I1122 07:30:56.449381 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:30:56.449393 3798194 net.go:659] Add eni metrics success, count: 7 E1122 07:30:57.450695 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 07:30:57.450708 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:30:57.450896 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:30:57.450905 3798194 disk_worker.go:723] disk space is not compliant W1122 07:30:57.450908 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:30:57.459215 3798194 disk_worker.go:509] system disk:vda1 I1122 07:30:57.459255 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 07:30:58.448465 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 07:30:58.448480 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 07:30:59.448977 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 07:30:59.450052 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:30:59.450118 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:30:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:30:59.471640 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:31:02.966150 3798194 disk_info.go:125] begin check local disk info of client I1122 07:31:02.971895 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:31:02.971906 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002eb580 0xc0002eb5c0] E1122 07:31:06.413247 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:31:06.413271 3798194 memory.go:184] no items to output this cycle I1122 07:31:06.413421 3798194 cpu.go:275] no items to output this cycle E1122 07:31:16.413255 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:31:16.413277 3798194 memory.go:184] no items to output this cycle I1122 07:31:16.413427 3798194 cpu.go:275] no items to output this cycle E1122 07:31:26.411916 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:31:26.411948 3798194 memory.go:191] Add success. I1122 07:31:26.412319 3798194 cpu.go:282] Add success. I1122 07:31:26.442029 3798194 net.go:642] Add success. I1122 07:31:26.446180 3798194 net.go:778] primary dev: ETH0 I1122 07:31:26.446196 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:31:26.446211 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:31:29.450004 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:31:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:31:29.450138 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:31:36.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:31:36.413252 3798194 memory.go:184] no items to output this cycle I1122 07:31:36.413396 3798194 cpu.go:275] no items to output this cycle E1122 07:31:46.412407 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:31:46.412437 3798194 memory.go:184] no items to output this cycle I1122 07:31:46.412979 3798194 cpu.go:275] no items to output this cycle E1122 07:31:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:31:56.412266 3798194 memory.go:191] Add success. W1122 07:31:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:31:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:31:56.412316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:31:56.416742 3798194 cpu.go:282] Add success. I1122 07:31:56.441144 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:31:56.441670 3798194 net.go:642] Add success. I1122 07:31:56.443147 3798194 net.go:778] primary dev: ETH0 I1122 07:31:56.443162 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:31:56.443174 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:31:57.451024 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:31:57.451216 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:31:57.451271 3798194 disk_worker.go:723] disk space is not compliant W1122 07:31:57.451275 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:31:57.465181 3798194 disk_worker.go:509] system disk:vda1 I1122 07:31:57.465224 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:31:58.446372 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:31:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:31:59.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:31:59.450500 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:31:59.471020 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:32:02.972013 3798194 disk_info.go:125] begin check local disk info of client I1122 07:32:02.975845 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:32:02.975854 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00025ef00 0xc00025ef40] E1122 07:32:06.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:32:06.411987 3798194 memory.go:184] no items to output this cycle I1122 07:32:06.412147 3798194 cpu.go:275] no items to output this cycle E1122 07:32:16.411973 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:32:16.411997 3798194 memory.go:184] no items to output this cycle I1122 07:32:16.412158 3798194 cpu.go:275] no items to output this cycle E1122 07:32:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:32:26.411951 3798194 memory.go:191] Add success. I1122 07:32:26.412311 3798194 cpu.go:282] Add success. I1122 07:32:26.442083 3798194 net.go:642] Add success. I1122 07:32:26.446502 3798194 net.go:778] primary dev: ETH0 I1122 07:32:26.446517 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:32:26.446528 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:32:29.450145 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:32:29.450227 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:32:29.450264 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:32:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:32:36.411979 3798194 memory.go:184] no items to output this cycle I1122 07:32:36.412132 3798194 cpu.go:275] no items to output this cycle E1122 07:32:46.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:32:46.413256 3798194 memory.go:184] no items to output this cycle I1122 07:32:46.413400 3798194 cpu.go:275] no items to output this cycle E1122 07:32:56.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:32:56.413266 3798194 memory.go:191] Add success. W1122 07:32:56.413302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:32:56.413314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:32:56.413317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:32:56.417510 3798194 cpu.go:282] Add success. I1122 07:32:56.441894 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:32:56.442516 3798194 net.go:642] Add success. I1122 07:32:56.444086 3798194 net.go:778] primary dev: ETH0 I1122 07:32:56.444100 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:32:56.444125 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:32:57.451297 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:32:57.451504 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:32:57.451516 3798194 disk_worker.go:723] disk space is not compliant W1122 07:32:57.451519 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:32:57.460567 3798194 disk_worker.go:509] system disk:vda1 I1122 07:32:57.460604 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:32:58.446287 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:32:59.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:32:59.450483 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:32:59.450509 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:32:59.471018 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:33:02.975961 3798194 disk_info.go:125] begin check local disk info of client I1122 07:33:02.979884 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:33:02.979893 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000491440 0xc000491480] E1122 07:33:06.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:33:06.411977 3798194 memory.go:184] no items to output this cycle I1122 07:33:06.412135 3798194 cpu.go:275] no items to output this cycle E1122 07:33:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:33:16.411982 3798194 memory.go:184] no items to output this cycle I1122 07:33:16.412144 3798194 cpu.go:275] no items to output this cycle I1122 07:33:17.657751 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:33:17.657760 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:33:26.411138 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:33:26.411174 3798194 memory.go:191] Add success. I1122 07:33:26.411510 3798194 cpu.go:282] Add success. I1122 07:33:26.442101 3798194 net.go:642] Add success. I1122 07:33:26.446197 3798194 net.go:778] primary dev: ETH0 I1122 07:33:26.446212 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:33:26.446225 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:33:29.450028 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:33:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:33:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:33:36.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:33:36.412261 3798194 memory.go:184] no items to output this cycle I1122 07:33:36.412406 3798194 cpu.go:275] no items to output this cycle E1122 07:33:46.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:33:46.413252 3798194 memory.go:184] no items to output this cycle I1122 07:33:46.413408 3798194 cpu.go:275] no items to output this cycle E1122 07:33:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:33:56.411954 3798194 memory.go:191] Add success. W1122 07:33:56.411982 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:33:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:33:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:33:56.412345 3798194 cpu.go:282] Add success. I1122 07:33:56.441823 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:33:56.442397 3798194 net.go:642] Add success. I1122 07:33:56.446863 3798194 net.go:778] primary dev: ETH0 I1122 07:33:56.446878 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:33:56.446890 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:33:57.451287 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:33:57.451487 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:33:57.451498 3798194 disk_worker.go:723] disk space is not compliant W1122 07:33:57.451501 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:33:57.459883 3798194 disk_worker.go:509] system disk:vda1 I1122 07:33:57.459927 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:33:58.446384 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:33:59.450493 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:33:59.450573 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:33:59.450599 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:33:59.471134 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:34:02.982138 3798194 disk_info.go:125] begin check local disk info of client I1122 07:34:02.985999 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:34:02.986009 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d82c0 0xc0003d8300] E1122 07:34:06.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:34:06.413267 3798194 memory.go:184] no items to output this cycle I1122 07:34:06.413411 3798194 cpu.go:275] no items to output this cycle E1122 07:34:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:34:16.413258 3798194 memory.go:184] no items to output this cycle I1122 07:34:16.413420 3798194 cpu.go:275] no items to output this cycle E1122 07:34:26.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:34:26.411947 3798194 memory.go:191] Add success. I1122 07:34:26.412304 3798194 cpu.go:282] Add success. I1122 07:34:26.441318 3798194 net.go:642] Add success. I1122 07:34:26.445412 3798194 net.go:778] primary dev: ETH0 I1122 07:34:26.445427 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:34:26.445440 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:34:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:34:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:34:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:34:36.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:34:36.412262 3798194 memory.go:184] no items to output this cycle I1122 07:34:36.412410 3798194 cpu.go:275] no items to output this cycle E1122 07:34:46.413099 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:34:46.413139 3798194 memory.go:184] no items to output this cycle I1122 07:34:46.413287 3798194 cpu.go:275] no items to output this cycle E1122 07:34:56.411929 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:34:56.411960 3798194 memory.go:191] Add success. W1122 07:34:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:34:56.412006 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:34:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:34:56.412335 3798194 cpu.go:282] Add success. I1122 07:34:56.441801 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:34:56.442410 3798194 net.go:642] Add success. I1122 07:34:56.446566 3798194 net.go:778] primary dev: ETH0 I1122 07:34:56.446581 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:34:56.446594 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:34:57.450853 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:34:57.451062 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:34:57.451074 3798194 disk_worker.go:723] disk space is not compliant W1122 07:34:57.451077 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:34:57.459996 3798194 disk_worker.go:509] system disk:vda1 I1122 07:34:57.460039 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:34:58.446388 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:34:59.450331 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:34:59.450411 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:34:59.450437 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:34:59.470981 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:35:02.986123 3798194 disk_info.go:125] begin check local disk info of client I1122 07:35:02.989926 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:35:02.989935 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002699c0 0xc000269a00] E1122 07:35:06.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:35:06.412257 3798194 memory.go:184] no items to output this cycle I1122 07:35:06.412411 3798194 cpu.go:275] no items to output this cycle E1122 07:35:16.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:35:16.413259 3798194 memory.go:184] no items to output this cycle I1122 07:35:16.413414 3798194 cpu.go:275] no items to output this cycle E1122 07:35:26.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:35:26.411944 3798194 memory.go:191] Add success. I1122 07:35:26.412295 3798194 cpu.go:282] Add success. I1122 07:35:26.441926 3798194 net.go:642] Add success. I1122 07:35:26.446691 3798194 net.go:778] primary dev: ETH0 I1122 07:35:26.446706 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:35:26.446721 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:35:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:35:29.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:35:29.450132 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:35:36.412079 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:35:36.412101 3798194 memory.go:184] no items to output this cycle I1122 07:35:36.412262 3798194 cpu.go:275] no items to output this cycle E1122 07:35:46.412225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:35:46.412247 3798194 memory.go:184] no items to output this cycle I1122 07:35:46.412387 3798194 cpu.go:275] no items to output this cycle E1122 07:35:56.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:35:56.413261 3798194 memory.go:191] Add success. W1122 07:35:56.413292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:35:56.413304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:35:56.413307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:35:56.417403 3798194 cpu.go:282] Add success. I1122 07:35:56.441209 3798194 net.go:642] Add success. I1122 07:35:56.443783 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795302356, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 07:35:56.443893 3798194 net.go:778] primary dev: ETH0 I1122 07:35:56.443907 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:35:56.443920 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:35:56.442851 3798194 event_worker.go:152] Polling the log file for events... I1122 07:35:56.443212 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:35:56.454754 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 07:35:57.449223 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 07:35:57.449236 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:35:57.449478 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:35:57.449487 3798194 disk_worker.go:723] disk space is not compliant W1122 07:35:57.449491 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:35:57.451289 3798194 disk_worker.go:509] system disk:vda1 I1122 07:35:57.451329 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 07:35:58.448511 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 07:35:58.448527 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 07:35:59.448989 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 07:35:59.450064 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:35:59.450122 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:35:59.450140 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:35:59.471664 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:36:02.991144 3798194 disk_info.go:125] begin check local disk info of client I1122 07:36:02.994958 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:36:02.994967 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030f640 0xc00030f680] E1122 07:36:06.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:36:06.413266 3798194 memory.go:184] no items to output this cycle I1122 07:36:06.413407 3798194 cpu.go:275] no items to output this cycle E1122 07:36:16.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:36:16.412264 3798194 memory.go:184] no items to output this cycle I1122 07:36:16.412411 3798194 cpu.go:275] no items to output this cycle I1122 07:36:17.658214 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:36:17.658223 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:36:26.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:36:26.413276 3798194 memory.go:191] Add success. I1122 07:36:26.417806 3798194 cpu.go:282] Add success. I1122 07:36:26.441632 3798194 net.go:642] Add success. I1122 07:36:26.443129 3798194 net.go:778] primary dev: ETH0 I1122 07:36:26.443145 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:36:26.443156 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:36:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:36:29.450487 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:36:29.450512 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:36:36.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:36:36.412873 3798194 memory.go:184] no items to output this cycle I1122 07:36:36.413037 3798194 cpu.go:275] no items to output this cycle E1122 07:36:46.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:36:46.413262 3798194 memory.go:184] no items to output this cycle I1122 07:36:46.413413 3798194 cpu.go:275] no items to output this cycle E1122 07:36:56.411932 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:36:56.411963 3798194 memory.go:191] Add success. W1122 07:36:56.411996 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:36:56.412009 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:36:56.412012 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:36:56.412352 3798194 cpu.go:282] Add success. I1122 07:36:56.441954 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:36:56.442540 3798194 net.go:642] Add success. I1122 07:36:56.446554 3798194 net.go:778] primary dev: ETH0 I1122 07:36:56.446569 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:36:56.446582 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:36:57.450597 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:36:57.450776 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:36:57.450833 3798194 disk_worker.go:723] disk space is not compliant W1122 07:36:57.450837 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:36:57.459183 3798194 disk_worker.go:509] system disk:vda1 I1122 07:36:57.459228 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:36:58.446069 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:36:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:36:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:36:59.450147 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:36:59.471670 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:37:02.995073 3798194 disk_info.go:125] begin check local disk info of client I1122 07:37:02.998975 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:37:02.998983 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004448c0 0xc000444900] E1122 07:37:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:37:06.411979 3798194 memory.go:184] no items to output this cycle I1122 07:37:06.412133 3798194 cpu.go:275] no items to output this cycle E1122 07:37:16.412865 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:37:16.412889 3798194 memory.go:184] no items to output this cycle I1122 07:37:16.413033 3798194 cpu.go:275] no items to output this cycle E1122 07:37:26.413137 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:37:26.413173 3798194 memory.go:191] Add success. I1122 07:37:26.417296 3798194 cpu.go:282] Add success. I1122 07:37:26.441802 3798194 net.go:642] Add success. I1122 07:37:26.443655 3798194 net.go:778] primary dev: ETH0 I1122 07:37:26.443669 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:37:26.443682 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:37:29.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:37:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:37:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:37:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:37:36.412252 3798194 memory.go:184] no items to output this cycle I1122 07:37:36.412393 3798194 cpu.go:275] no items to output this cycle E1122 07:37:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:37:46.411983 3798194 memory.go:184] no items to output this cycle I1122 07:37:46.412145 3798194 cpu.go:275] no items to output this cycle E1122 07:37:56.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:37:56.411955 3798194 memory.go:191] Add success. W1122 07:37:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:37:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:37:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:37:56.412363 3798194 cpu.go:282] Add success. I1122 07:37:56.441810 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:37:56.442333 3798194 net.go:642] Add success. I1122 07:37:56.445824 3798194 net.go:778] primary dev: ETH0 I1122 07:37:56.445841 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:37:56.445853 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:37:57.450633 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:37:57.450790 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:37:57.450842 3798194 disk_worker.go:723] disk space is not compliant W1122 07:37:57.450846 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:37:57.459242 3798194 disk_worker.go:509] system disk:vda1 I1122 07:37:57.459285 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:37:58.446008 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:37:59.450118 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:37:59.450195 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:37:59.450222 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:37:59.470689 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:38:03.001713 3798194 disk_info.go:125] begin check local disk info of client I1122 07:38:03.010168 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:38:03.010179 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003698c0 0xc000369900] E1122 07:38:06.412100 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:38:06.412135 3798194 memory.go:184] no items to output this cycle I1122 07:38:06.412283 3798194 cpu.go:275] no items to output this cycle E1122 07:38:16.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:38:16.412264 3798194 memory.go:184] no items to output this cycle I1122 07:38:16.412408 3798194 cpu.go:275] no items to output this cycle E1122 07:38:26.411907 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:38:26.411939 3798194 memory.go:191] Add success. I1122 07:38:26.412304 3798194 cpu.go:282] Add success. I1122 07:38:26.442139 3798194 net.go:642] Add success. I1122 07:38:26.446422 3798194 net.go:778] primary dev: ETH0 I1122 07:38:26.446440 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:38:26.446461 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:38:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:38:29.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:38:29.450149 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:38:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:38:36.413260 3798194 memory.go:184] no items to output this cycle I1122 07:38:36.413403 3798194 cpu.go:275] no items to output this cycle E1122 07:38:46.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:38:46.411984 3798194 memory.go:184] no items to output this cycle I1122 07:38:46.412139 3798194 cpu.go:275] no items to output this cycle E1122 07:38:56.411240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:38:56.411274 3798194 memory.go:191] Add success. W1122 07:38:56.411307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:38:56.411318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:38:56.411322 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:38:56.415680 3798194 cpu.go:282] Add success. I1122 07:38:56.441239 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:38:56.441730 3798194 net.go:642] Add success. I1122 07:38:56.443232 3798194 net.go:778] primary dev: ETH0 I1122 07:38:56.443246 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:38:56.443260 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:38:57.450739 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:38:57.450949 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:38:57.450960 3798194 disk_worker.go:723] disk space is not compliant W1122 07:38:57.450965 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:38:57.459864 3798194 disk_worker.go:509] system disk:vda1 I1122 07:38:57.459910 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:38:58.446386 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:38:59.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:38:59.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:38:59.450121 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:38:59.471643 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:39:03.010284 3798194 disk_info.go:125] begin check local disk info of client I1122 07:39:03.014209 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:39:03.014219 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003cdd40 0xc0003cdd80] E1122 07:39:06.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:39:06.411984 3798194 memory.go:184] no items to output this cycle I1122 07:39:06.412158 3798194 cpu.go:275] no items to output this cycle E1122 07:39:16.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:39:16.413252 3798194 memory.go:184] no items to output this cycle I1122 07:39:16.413404 3798194 cpu.go:275] no items to output this cycle I1122 07:39:17.658383 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:39:17.658392 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:39:26.412825 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:39:26.412860 3798194 memory.go:191] Add success. I1122 07:39:26.417307 3798194 cpu.go:282] Add success. I1122 07:39:26.441744 3798194 net.go:642] Add success. I1122 07:39:26.443348 3798194 net.go:778] primary dev: ETH0 I1122 07:39:26.443362 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:39:26.443375 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:39:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:39:29.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:39:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:39:36.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:39:36.412249 3798194 memory.go:184] no items to output this cycle I1122 07:39:36.412401 3798194 cpu.go:275] no items to output this cycle E1122 07:39:46.413317 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:39:46.413341 3798194 memory.go:184] no items to output this cycle I1122 07:39:46.413485 3798194 cpu.go:275] no items to output this cycle E1122 07:39:56.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:39:56.412889 3798194 memory.go:191] Add success. W1122 07:39:56.412918 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:39:56.412930 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:39:56.412933 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:39:56.417300 3798194 cpu.go:282] Add success. I1122 07:39:56.441804 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:39:56.442905 3798194 net.go:642] Add success. I1122 07:39:56.445777 3798194 net.go:778] primary dev: ETH0 I1122 07:39:56.445794 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:39:56.445808 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:39:57.450374 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:39:57.450571 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:39:57.450581 3798194 disk_worker.go:723] disk space is not compliant W1122 07:39:57.450585 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:39:57.458870 3798194 disk_worker.go:509] system disk:vda1 I1122 07:39:57.458909 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:39:58.446379 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:39:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:39:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:39:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:39:59.471671 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:40:03.014324 3798194 disk_info.go:125] begin check local disk info of client I1122 07:40:03.018164 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:40:03.018172 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004f3a00 0xc0004f3a40] E1122 07:40:06.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:40:06.412879 3798194 memory.go:184] no items to output this cycle I1122 07:40:06.413030 3798194 cpu.go:275] no items to output this cycle E1122 07:40:16.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:40:16.411988 3798194 memory.go:184] no items to output this cycle I1122 07:40:16.412149 3798194 cpu.go:275] no items to output this cycle E1122 07:40:26.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:40:26.411945 3798194 memory.go:191] Add success. I1122 07:40:26.412296 3798194 cpu.go:282] Add success. I1122 07:40:26.441918 3798194 net.go:642] Add success. I1122 07:40:26.446206 3798194 net.go:778] primary dev: ETH0 I1122 07:40:26.446222 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:40:26.446235 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:40:29.450065 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:40:29.450165 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:40:29.450193 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:40:36.413168 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:40:36.413191 3798194 memory.go:184] no items to output this cycle I1122 07:40:36.413331 3798194 cpu.go:275] no items to output this cycle E1122 07:40:46.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:40:46.412262 3798194 memory.go:184] no items to output this cycle I1122 07:40:46.412411 3798194 cpu.go:275] no items to output this cycle E1122 07:40:56.411908 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:40:56.411940 3798194 memory.go:191] Add success. W1122 07:40:56.411968 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:40:56.411979 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:40:56.411983 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:40:56.412314 3798194 cpu.go:282] Add success. I1122 07:40:56.444621 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:40:56.445363 3798194 net.go:642] Add success. I1122 07:40:56.449540 3798194 net.go:778] primary dev: ETH0 I1122 07:40:56.449567 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:40:56.449586 3798194 net.go:659] Add eni metrics success, count: 7 E1122 07:40:57.451093 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 07:40:57.451107 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:40:57.451335 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:40:57.451345 3798194 disk_worker.go:723] disk space is not compliant W1122 07:40:57.451348 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:40:57.460073 3798194 disk_worker.go:509] system disk:vda1 I1122 07:40:57.460132 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 07:40:58.448465 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 07:40:58.448479 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 07:40:59.449199 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 07:40:59.450273 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:40:59.450326 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:40:59.450345 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:40:59.470878 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:41:03.020144 3798194 disk_info.go:125] begin check local disk info of client I1122 07:41:03.023967 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:41:03.023975 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004f32c0 0xc0004f3300] E1122 07:41:06.412766 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:41:06.412789 3798194 memory.go:184] no items to output this cycle I1122 07:41:06.412938 3798194 cpu.go:275] no items to output this cycle E1122 07:41:16.412333 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:41:16.412354 3798194 memory.go:184] no items to output this cycle I1122 07:41:16.412518 3798194 cpu.go:275] no items to output this cycle E1122 07:41:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:41:26.413265 3798194 memory.go:191] Add success. I1122 07:41:26.417573 3798194 cpu.go:282] Add success. I1122 07:41:26.442132 3798194 net.go:642] Add success. I1122 07:41:26.443540 3798194 net.go:778] primary dev: ETH0 I1122 07:41:26.443554 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:41:26.443568 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:41:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:41:29.450482 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:41:29.450508 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:41:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:41:36.413259 3798194 memory.go:184] no items to output this cycle I1122 07:41:36.413401 3798194 cpu.go:275] no items to output this cycle E1122 07:41:46.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:41:46.412886 3798194 memory.go:184] no items to output this cycle I1122 07:41:46.413031 3798194 cpu.go:275] no items to output this cycle E1122 07:41:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:41:56.411951 3798194 memory.go:191] Add success. W1122 07:41:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:41:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:41:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:41:56.412349 3798194 cpu.go:282] Add success. I1122 07:41:56.441803 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:41:56.442441 3798194 net.go:642] Add success. I1122 07:41:56.446807 3798194 net.go:778] primary dev: ETH0 I1122 07:41:56.446823 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:41:56.446835 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:41:57.450618 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:41:57.450958 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:41:57.450970 3798194 disk_worker.go:723] disk space is not compliant W1122 07:41:57.450973 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:41:57.460056 3798194 disk_worker.go:509] system disk:vda1 I1122 07:41:57.460096 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:41:58.446005 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:41:59.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:41:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:41:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:41:59.471699 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:42:03.026844 3798194 disk_info.go:125] begin check local disk info of client I1122 07:42:03.030723 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:42:03.030733 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004399c0 0xc000439a00] E1122 07:42:06.412952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:42:06.412974 3798194 memory.go:184] no items to output this cycle I1122 07:42:06.413142 3798194 cpu.go:275] no items to output this cycle E1122 07:42:16.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:42:16.412256 3798194 memory.go:184] no items to output this cycle I1122 07:42:16.412399 3798194 cpu.go:275] no items to output this cycle I1122 07:42:17.658550 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:42:17.658559 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:42:26.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:42:26.412275 3798194 memory.go:191] Add success. I1122 07:42:26.416597 3798194 cpu.go:282] Add success. I1122 07:42:26.442196 3798194 net.go:642] Add success. I1122 07:42:26.443635 3798194 net.go:778] primary dev: ETH0 I1122 07:42:26.443650 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:42:26.443663 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:42:29.450437 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:42:29.450526 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:42:29.450565 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:42:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:42:36.413258 3798194 memory.go:184] no items to output this cycle I1122 07:42:36.413395 3798194 cpu.go:275] no items to output this cycle E1122 07:42:46.413054 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:42:46.413077 3798194 memory.go:184] no items to output this cycle I1122 07:42:46.413244 3798194 cpu.go:275] no items to output this cycle E1122 07:42:56.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:42:56.411950 3798194 memory.go:191] Add success. W1122 07:42:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:42:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:42:56.411994 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:42:56.412324 3798194 cpu.go:282] Add success. I1122 07:42:56.440962 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:42:56.441580 3798194 net.go:642] Add success. I1122 07:42:56.445930 3798194 net.go:778] primary dev: ETH0 I1122 07:42:56.445946 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:42:56.445957 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:42:57.450601 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:42:57.450799 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:42:57.450810 3798194 disk_worker.go:723] disk space is not compliant W1122 07:42:57.450814 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:42:57.459163 3798194 disk_worker.go:509] system disk:vda1 I1122 07:42:57.459334 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:42:58.446013 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:42:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:42:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:42:59.450139 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:42:59.471675 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:43:03.030837 3798194 disk_info.go:125] begin check local disk info of client I1122 07:43:03.034674 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:43:03.034683 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b4cc0 0xc0003b4d00] E1122 07:43:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:43:06.411983 3798194 memory.go:184] no items to output this cycle I1122 07:43:06.412140 3798194 cpu.go:275] no items to output this cycle E1122 07:43:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:43:16.411977 3798194 memory.go:184] no items to output this cycle I1122 07:43:16.412128 3798194 cpu.go:275] no items to output this cycle E1122 07:43:26.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:43:26.412262 3798194 memory.go:191] Add success. I1122 07:43:26.416484 3798194 cpu.go:282] Add success. I1122 07:43:26.442153 3798194 net.go:642] Add success. I1122 07:43:26.443620 3798194 net.go:778] primary dev: ETH0 I1122 07:43:26.443643 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:43:26.443658 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:43:29.449941 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:43:29.450013 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:43:29.450039 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:43:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:43:36.411980 3798194 memory.go:184] no items to output this cycle I1122 07:43:36.412143 3798194 cpu.go:275] no items to output this cycle E1122 07:43:46.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:43:46.413250 3798194 memory.go:184] no items to output this cycle I1122 07:43:46.413397 3798194 cpu.go:275] no items to output this cycle E1122 07:43:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:43:56.411956 3798194 memory.go:191] Add success. W1122 07:43:56.411983 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:43:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:43:56.411998 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:43:56.412350 3798194 cpu.go:282] Add success. I1122 07:43:56.440976 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:43:56.441565 3798194 net.go:642] Add success. I1122 07:43:56.445365 3798194 net.go:778] primary dev: ETH0 I1122 07:43:56.445380 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:43:56.445391 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:43:57.450513 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:43:57.450672 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:43:57.450723 3798194 disk_worker.go:723] disk space is not compliant W1122 07:43:57.450726 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:43:57.459037 3798194 disk_worker.go:509] system disk:vda1 I1122 07:43:57.459077 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:43:58.446590 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:43:59.449641 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:43:59.449741 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:43:59.449768 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:43:59.471332 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:44:03.036170 3798194 disk_info.go:125] begin check local disk info of client I1122 07:44:03.041600 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:44:03.041612 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fe280 0xc0003fe2c0] E1122 07:44:06.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:44:06.411973 3798194 memory.go:184] no items to output this cycle I1122 07:44:06.412282 3798194 cpu.go:275] no items to output this cycle E1122 07:44:16.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:44:16.411967 3798194 memory.go:184] no items to output this cycle I1122 07:44:16.412126 3798194 cpu.go:275] no items to output this cycle E1122 07:44:26.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:44:26.411945 3798194 memory.go:191] Add success. I1122 07:44:26.412324 3798194 cpu.go:282] Add success. I1122 07:44:26.442086 3798194 net.go:642] Add success. I1122 07:44:26.446030 3798194 net.go:778] primary dev: ETH0 I1122 07:44:26.446046 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:44:26.446059 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:44:29.449980 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:44:29.450063 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:44:29.450094 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:44:36.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:44:36.412267 3798194 memory.go:184] no items to output this cycle I1122 07:44:36.412415 3798194 cpu.go:275] no items to output this cycle E1122 07:44:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:44:46.412257 3798194 memory.go:184] no items to output this cycle I1122 07:44:46.412402 3798194 cpu.go:275] no items to output this cycle E1122 07:44:56.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:44:56.413265 3798194 memory.go:191] Add success. W1122 07:44:56.413300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:44:56.413314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:44:56.413317 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:44:56.417926 3798194 cpu.go:282] Add success. I1122 07:44:56.441338 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:44:56.441863 3798194 net.go:642] Add success. I1122 07:44:56.443406 3798194 net.go:778] primary dev: ETH0 I1122 07:44:56.443422 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:44:56.443434 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:44:57.445482 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:44:57.445692 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:44:57.445753 3798194 disk_worker.go:723] disk space is not compliant W1122 07:44:57.445756 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:44:57.447834 3798194 disk_worker.go:509] system disk:vda1 I1122 07:44:57.447877 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:44:58.446378 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:44:59.450388 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:44:59.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:44:59.450503 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:44:59.471012 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:45:03.043146 3798194 disk_info.go:125] begin check local disk info of client I1122 07:45:03.046985 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:45:03.046994 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00035ef80 0xc00035efc0] E1122 07:45:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:45:06.411983 3798194 memory.go:184] no items to output this cycle I1122 07:45:06.412137 3798194 cpu.go:275] no items to output this cycle E1122 07:45:16.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:45:16.413254 3798194 memory.go:184] no items to output this cycle I1122 07:45:16.413528 3798194 cpu.go:275] no items to output this cycle I1122 07:45:17.658714 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:45:17.658722 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:45:26.411164 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:45:26.411196 3798194 memory.go:191] Add success. I1122 07:45:26.411562 3798194 cpu.go:282] Add success. I1122 07:45:26.442189 3798194 net.go:642] Add success. I1122 07:45:26.446326 3798194 net.go:778] primary dev: ETH0 I1122 07:45:26.446343 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:45:26.446356 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:45:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:45:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:45:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:45:36.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:45:36.412249 3798194 memory.go:184] no items to output this cycle I1122 07:45:36.412396 3798194 cpu.go:275] no items to output this cycle E1122 07:45:46.412341 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:45:46.412365 3798194 memory.go:184] no items to output this cycle I1122 07:45:46.412513 3798194 cpu.go:275] no items to output this cycle E1122 07:45:56.412224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:45:56.412254 3798194 memory.go:191] Add success. W1122 07:45:56.412281 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:45:56.412292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:45:56.412296 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:45:56.417328 3798194 cpu.go:282] Add success. I1122 07:45:56.441653 3798194 event_worker.go:152] Polling the log file for events... I1122 07:45:56.444769 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:45:56.445276 3798194 net.go:642] Add success. I1122 07:45:56.446758 3798194 net.go:778] primary dev: ETH0 I1122 07:45:56.446773 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:45:56.446786 3798194 net.go:659] Add eni metrics success, count: 7 E1122 07:45:57.450931 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 07:45:57.450945 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:45:57.451164 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:45:57.451175 3798194 disk_worker.go:723] disk space is not compliant W1122 07:45:57.451178 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:45:57.459875 3798194 disk_worker.go:509] system disk:vda1 I1122 07:45:57.459916 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 07:45:58.448104 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 07:45:58.448133 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 07:45:59.448667 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 07:45:59.449741 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:45:59.449791 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:45:59.449813 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:45:59.471330 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:46:03.047100 3798194 disk_info.go:125] begin check local disk info of client I1122 07:46:03.051046 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:46:03.051056 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030de40 0xc00030de80] E1122 07:46:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:46:06.411979 3798194 memory.go:184] no items to output this cycle I1122 07:46:06.412138 3798194 cpu.go:275] no items to output this cycle E1122 07:46:16.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:46:16.413257 3798194 memory.go:184] no items to output this cycle I1122 07:46:16.413401 3798194 cpu.go:275] no items to output this cycle E1122 07:46:26.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:46:26.413272 3798194 memory.go:191] Add success. I1122 07:46:26.417425 3798194 cpu.go:282] Add success. I1122 07:46:26.442170 3798194 net.go:642] Add success. I1122 07:46:26.443614 3798194 net.go:778] primary dev: ETH0 I1122 07:46:26.443627 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:46:26.443640 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:46:29.450463 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:46:29.450538 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:46:29.450564 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:46:36.413299 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:46:36.413323 3798194 memory.go:184] no items to output this cycle I1122 07:46:36.413478 3798194 cpu.go:275] no items to output this cycle E1122 07:46:46.412879 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:46:46.412901 3798194 memory.go:184] no items to output this cycle I1122 07:46:46.413077 3798194 cpu.go:275] no items to output this cycle E1122 07:46:56.411930 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:46:56.411968 3798194 memory.go:191] Add success. W1122 07:46:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:46:56.412006 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:46:56.412010 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:46:56.412368 3798194 cpu.go:282] Add success. I1122 07:46:56.441943 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:46:56.442612 3798194 net.go:642] Add success. I1122 07:46:56.446651 3798194 net.go:778] primary dev: ETH0 I1122 07:46:56.446667 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:46:56.446680 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:46:57.450549 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:46:57.450747 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:46:57.450757 3798194 disk_worker.go:723] disk space is not compliant W1122 07:46:57.450760 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:46:57.459162 3798194 disk_worker.go:509] system disk:vda1 I1122 07:46:57.459202 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:46:58.446199 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:46:59.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:46:59.450099 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:46:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:46:59.471643 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:47:03.053149 3798194 disk_info.go:125] begin check local disk info of client I1122 07:47:03.056986 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:47:03.056996 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e17c0 0xc0002e1800] E1122 07:47:06.413283 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:47:06.413306 3798194 memory.go:184] no items to output this cycle I1122 07:47:06.413453 3798194 cpu.go:275] no items to output this cycle E1122 07:47:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:47:16.413258 3798194 memory.go:184] no items to output this cycle I1122 07:47:16.413404 3798194 cpu.go:275] no items to output this cycle E1122 07:47:26.412868 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:47:26.412903 3798194 memory.go:191] Add success. I1122 07:47:26.417393 3798194 cpu.go:282] Add success. I1122 07:47:26.441917 3798194 net.go:642] Add success. I1122 07:47:26.443457 3798194 net.go:778] primary dev: ETH0 I1122 07:47:26.443472 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:47:26.443484 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:47:29.449889 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:47:29.449971 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:47:29.449996 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:47:36.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:47:36.411988 3798194 memory.go:184] no items to output this cycle I1122 07:47:36.412146 3798194 cpu.go:275] no items to output this cycle E1122 07:47:46.413408 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:47:46.413431 3798194 memory.go:184] no items to output this cycle I1122 07:47:46.413575 3798194 cpu.go:275] no items to output this cycle E1122 07:47:56.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:47:56.411945 3798194 memory.go:191] Add success. W1122 07:47:56.411981 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:47:56.411993 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:47:56.411997 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:47:56.412344 3798194 cpu.go:282] Add success. I1122 07:47:56.441796 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:47:56.442388 3798194 net.go:642] Add success. I1122 07:47:56.446587 3798194 net.go:778] primary dev: ETH0 I1122 07:47:56.446602 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:47:56.446616 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:47:57.450357 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:47:57.450565 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:47:57.450576 3798194 disk_worker.go:723] disk space is not compliant W1122 07:47:57.450579 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:47:57.458620 3798194 disk_worker.go:509] system disk:vda1 I1122 07:47:57.458661 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:47:58.446031 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:47:59.450031 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:47:59.450124 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:47:59.450162 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:47:59.471694 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:48:03.057120 3798194 disk_info.go:125] begin check local disk info of client I1122 07:48:03.060648 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:48:03.060656 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046e1c0 0xc00046e200] E1122 07:48:06.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:48:06.413263 3798194 memory.go:184] no items to output this cycle I1122 07:48:06.413405 3798194 cpu.go:275] no items to output this cycle E1122 07:48:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:48:16.411985 3798194 memory.go:184] no items to output this cycle I1122 07:48:16.412141 3798194 cpu.go:275] no items to output this cycle I1122 07:48:17.658878 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:48:17.658886 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:48:26.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:48:26.413283 3798194 memory.go:191] Add success. I1122 07:48:26.417605 3798194 cpu.go:282] Add success. I1122 07:48:26.442165 3798194 net.go:642] Add success. I1122 07:48:26.443556 3798194 net.go:778] primary dev: ETH0 I1122 07:48:26.443570 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:48:26.443582 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:48:29.450416 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:48:29.450491 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:48:29.450517 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:48:36.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:48:36.411991 3798194 memory.go:184] no items to output this cycle I1122 07:48:36.412145 3798194 cpu.go:275] no items to output this cycle E1122 07:48:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:48:46.412254 3798194 memory.go:184] no items to output this cycle I1122 07:48:46.412402 3798194 cpu.go:275] no items to output this cycle E1122 07:48:56.411911 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:48:56.411946 3798194 memory.go:191] Add success. W1122 07:48:56.411977 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:48:56.411989 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:48:56.411992 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:48:56.412333 3798194 cpu.go:282] Add success. I1122 07:48:56.441911 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:48:56.442487 3798194 net.go:642] Add success. I1122 07:48:56.446708 3798194 net.go:778] primary dev: ETH0 I1122 07:48:56.446723 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:48:56.446735 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:48:57.451042 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:48:57.451286 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:48:57.451297 3798194 disk_worker.go:723] disk space is not compliant W1122 07:48:57.451300 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:48:57.460014 3798194 disk_worker.go:509] system disk:vda1 I1122 07:48:57.460056 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:48:58.445991 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:48:59.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:48:59.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:48:59.450502 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:48:59.471009 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:49:03.063394 3798194 disk_info.go:125] begin check local disk info of client I1122 07:49:03.067263 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:49:03.067272 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000383840 0xc000383880] E1122 07:49:06.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:49:06.413257 3798194 memory.go:184] no items to output this cycle I1122 07:49:06.413399 3798194 cpu.go:275] no items to output this cycle E1122 07:49:16.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:49:16.411974 3798194 memory.go:184] no items to output this cycle I1122 07:49:16.412144 3798194 cpu.go:275] no items to output this cycle E1122 07:49:26.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:49:26.411954 3798194 memory.go:191] Add success. I1122 07:49:26.412315 3798194 cpu.go:282] Add success. I1122 07:49:26.441966 3798194 net.go:642] Add success. I1122 07:49:26.446174 3798194 net.go:778] primary dev: ETH0 I1122 07:49:26.446190 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:49:26.446204 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:49:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:49:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:49:29.450131 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:49:36.412249 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:49:36.412273 3798194 memory.go:184] no items to output this cycle I1122 07:49:36.412414 3798194 cpu.go:275] no items to output this cycle E1122 07:49:46.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:49:46.412887 3798194 memory.go:184] no items to output this cycle I1122 07:49:46.413034 3798194 cpu.go:275] no items to output this cycle E1122 07:49:56.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:49:56.412891 3798194 memory.go:191] Add success. W1122 07:49:56.412925 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:49:56.412936 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:49:56.412939 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:49:56.417216 3798194 cpu.go:282] Add success. I1122 07:49:56.441589 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:49:56.442059 3798194 net.go:642] Add success. I1122 07:49:56.443753 3798194 net.go:778] primary dev: ETH0 I1122 07:49:56.443767 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:49:56.443779 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:49:57.450697 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:49:57.450856 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:49:57.450908 3798194 disk_worker.go:723] disk space is not compliant W1122 07:49:57.450911 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:49:57.459259 3798194 disk_worker.go:509] system disk:vda1 I1122 07:49:57.459302 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:49:58.446370 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:49:59.450465 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:49:59.450544 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:49:59.450570 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:49:59.471086 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:50:03.067374 3798194 disk_info.go:125] begin check local disk info of client I1122 07:50:03.071245 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:50:03.071255 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f2e40 0xc0003f2e80] E1122 07:50:06.412227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:50:06.412251 3798194 memory.go:184] no items to output this cycle I1122 07:50:06.412394 3798194 cpu.go:275] no items to output this cycle E1122 07:50:16.413353 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:50:16.413374 3798194 memory.go:184] no items to output this cycle I1122 07:50:16.413535 3798194 cpu.go:275] no items to output this cycle E1122 07:50:26.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:50:26.411958 3798194 memory.go:191] Add success. I1122 07:50:26.412307 3798194 cpu.go:282] Add success. I1122 07:50:26.441931 3798194 net.go:642] Add success. I1122 07:50:26.446319 3798194 net.go:778] primary dev: ETH0 I1122 07:50:26.446334 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:50:26.446348 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:50:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:50:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:50:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:50:36.412242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:50:36.412271 3798194 memory.go:184] no items to output this cycle I1122 07:50:36.412415 3798194 cpu.go:275] no items to output this cycle E1122 07:50:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:50:46.411972 3798194 memory.go:184] no items to output this cycle I1122 07:50:46.412123 3798194 cpu.go:275] no items to output this cycle E1122 07:50:56.411463 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:50:56.411496 3798194 memory.go:191] Add success. W1122 07:50:56.411526 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:50:56.411537 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:50:56.411542 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:50:56.415718 3798194 cpu.go:282] Add success. I1122 07:50:56.444002 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:50:56.444527 3798194 net.go:642] Add success. I1122 07:50:56.446228 3798194 net.go:778] primary dev: ETH0 I1122 07:50:56.446242 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:50:56.446255 3798194 net.go:659] Add eni metrics success, count: 7 E1122 07:50:57.450656 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 07:50:57.450671 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:50:57.451173 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:50:57.451185 3798194 disk_worker.go:723] disk space is not compliant W1122 07:50:57.451189 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:50:57.460048 3798194 disk_worker.go:509] system disk:vda1 I1122 07:50:57.460089 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 07:50:58.447770 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 07:50:58.447783 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 07:50:59.448926 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 07:50:59.450001 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:50:59.450050 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:50:59.450068 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:50:59.471581 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:51:03.071359 3798194 disk_info.go:125] begin check local disk info of client I1122 07:51:03.075306 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:51:03.075318 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004cdb00 0xc0004cdb40] E1122 07:51:06.411981 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:51:06.412005 3798194 memory.go:184] no items to output this cycle I1122 07:51:06.412173 3798194 cpu.go:275] no items to output this cycle E1122 07:51:16.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:51:16.412882 3798194 memory.go:184] no items to output this cycle I1122 07:51:16.413042 3798194 cpu.go:275] no items to output this cycle I1122 07:51:17.659208 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:51:17.659218 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:51:26.411313 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:51:26.411346 3798194 memory.go:191] Add success. I1122 07:51:26.411689 3798194 cpu.go:282] Add success. I1122 07:51:26.441325 3798194 net.go:642] Add success. I1122 07:51:26.445336 3798194 net.go:778] primary dev: ETH0 I1122 07:51:26.445354 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:51:26.445366 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:51:29.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:51:29.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:51:29.450489 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:51:36.412954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:51:36.412980 3798194 memory.go:184] no items to output this cycle I1122 07:51:36.413142 3798194 cpu.go:275] no items to output this cycle E1122 07:51:46.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:51:46.411974 3798194 memory.go:184] no items to output this cycle I1122 07:51:46.412144 3798194 cpu.go:275] no items to output this cycle E1122 07:51:56.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:51:56.413271 3798194 memory.go:191] Add success. W1122 07:51:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:51:56.413315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:51:56.413318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:51:56.418010 3798194 cpu.go:282] Add success. I1122 07:51:56.441440 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:51:56.441992 3798194 net.go:642] Add success. I1122 07:51:56.443581 3798194 net.go:778] primary dev: ETH0 I1122 07:51:56.443597 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:51:56.443609 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:51:57.446457 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:51:57.446727 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:51:57.446740 3798194 disk_worker.go:723] disk space is not compliant W1122 07:51:57.446743 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:51:57.449526 3798194 disk_worker.go:509] system disk:vda1 I1122 07:51:57.449567 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:51:58.446373 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:51:59.450389 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:51:59.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:51:59.450496 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:51:59.471008 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:52:03.075420 3798194 disk_info.go:125] begin check local disk info of client I1122 07:52:03.079292 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:52:03.079301 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000336500 0xc000336540] E1122 07:52:06.413537 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:52:06.413558 3798194 memory.go:184] no items to output this cycle I1122 07:52:06.413723 3798194 cpu.go:275] no items to output this cycle E1122 07:52:16.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:52:16.412888 3798194 memory.go:184] no items to output this cycle I1122 07:52:16.413036 3798194 cpu.go:275] no items to output this cycle E1122 07:52:26.412847 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:52:26.412880 3798194 memory.go:191] Add success. I1122 07:52:26.417390 3798194 cpu.go:282] Add success. I1122 07:52:26.442100 3798194 net.go:642] Add success. I1122 07:52:26.443710 3798194 net.go:778] primary dev: ETH0 I1122 07:52:26.443724 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:52:26.443737 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:52:29.453177 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:52:29.453260 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:52:29.453294 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:52:36.411965 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:52:36.411989 3798194 memory.go:184] no items to output this cycle I1122 07:52:36.412152 3798194 cpu.go:275] no items to output this cycle E1122 07:52:46.412795 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:52:46.412819 3798194 memory.go:184] no items to output this cycle I1122 07:52:46.412968 3798194 cpu.go:275] no items to output this cycle E1122 07:52:56.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:52:56.413278 3798194 memory.go:191] Add success. W1122 07:52:56.413307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:52:56.413318 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:52:56.413322 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:52:56.417677 3798194 cpu.go:282] Add success. I1122 07:52:56.441066 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:52:56.441548 3798194 net.go:642] Add success. I1122 07:52:56.443256 3798194 net.go:778] primary dev: ETH0 I1122 07:52:56.443270 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:52:56.443282 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:52:57.450897 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:52:57.451066 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:52:57.451138 3798194 disk_worker.go:723] disk space is not compliant W1122 07:52:57.451142 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:52:57.462681 3798194 disk_worker.go:509] system disk:vda1 I1122 07:52:57.462724 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:52:58.445880 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:52:59.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:52:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:52:59.450507 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:52:59.471024 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:53:03.081894 3798194 disk_info.go:125] begin check local disk info of client I1122 07:53:03.086534 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:53:03.086544 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003cc000 0xc0003cc040] E1122 07:53:06.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:53:06.412888 3798194 memory.go:184] no items to output this cycle I1122 07:53:06.413044 3798194 cpu.go:275] no items to output this cycle E1122 07:53:16.411964 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:53:16.411988 3798194 memory.go:184] no items to output this cycle I1122 07:53:16.412145 3798194 cpu.go:275] no items to output this cycle E1122 07:53:26.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:53:26.411952 3798194 memory.go:191] Add success. I1122 07:53:26.412334 3798194 cpu.go:282] Add success. I1122 07:53:26.442042 3798194 net.go:642] Add success. I1122 07:53:26.446238 3798194 net.go:778] primary dev: ETH0 I1122 07:53:26.446253 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:53:26.446268 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:53:29.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:53:29.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:53:29.450505 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:53:36.412085 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:53:36.412108 3798194 memory.go:184] no items to output this cycle I1122 07:53:36.412267 3798194 cpu.go:275] no items to output this cycle E1122 07:53:46.411967 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:53:46.411988 3798194 memory.go:184] no items to output this cycle I1122 07:53:46.412147 3798194 cpu.go:275] no items to output this cycle E1122 07:53:56.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:53:56.411955 3798194 memory.go:191] Add success. W1122 07:53:56.411987 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:53:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:53:56.412004 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:53:56.412363 3798194 cpu.go:282] Add success. I1122 07:53:56.441832 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:53:56.442431 3798194 net.go:642] Add success. I1122 07:53:56.446450 3798194 net.go:778] primary dev: ETH0 I1122 07:53:56.446465 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:53:56.446477 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:53:57.450741 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:53:57.450901 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:53:57.450954 3798194 disk_worker.go:723] disk space is not compliant W1122 07:53:57.450957 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:53:57.459325 3798194 disk_worker.go:509] system disk:vda1 I1122 07:53:57.459367 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:53:58.446384 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:53:59.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:53:59.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:53:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:53:59.471666 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:54:03.089139 3798194 disk_info.go:125] begin check local disk info of client I1122 07:54:03.093002 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:54:03.093015 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000371480 0xc0003714c0] E1122 07:54:06.413176 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:54:06.413198 3798194 memory.go:184] no items to output this cycle I1122 07:54:06.413346 3798194 cpu.go:275] no items to output this cycle E1122 07:54:16.412845 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:54:16.412872 3798194 memory.go:184] no items to output this cycle I1122 07:54:16.413017 3798194 cpu.go:275] no items to output this cycle I1122 07:54:17.660217 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:54:17.660225 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:54:26.411329 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:54:26.411361 3798194 memory.go:191] Add success. I1122 07:54:26.411728 3798194 cpu.go:282] Add success. I1122 07:54:26.441300 3798194 net.go:642] Add success. I1122 07:54:26.445437 3798194 net.go:778] primary dev: ETH0 I1122 07:54:26.445452 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:54:26.445466 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:54:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:54:29.450092 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:54:29.450135 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:54:36.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:54:36.413256 3798194 memory.go:184] no items to output this cycle I1122 07:54:36.413400 3798194 cpu.go:275] no items to output this cycle E1122 07:54:46.412767 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:54:46.412790 3798194 memory.go:184] no items to output this cycle I1122 07:54:46.412939 3798194 cpu.go:275] no items to output this cycle E1122 07:54:56.412848 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:54:56.412884 3798194 memory.go:191] Add success. W1122 07:54:56.412917 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:54:56.412933 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:54:56.412936 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:54:56.417392 3798194 cpu.go:282] Add success. I1122 07:54:56.441796 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:54:56.442349 3798194 net.go:642] Add success. I1122 07:54:56.443946 3798194 net.go:778] primary dev: ETH0 I1122 07:54:56.443961 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:54:56.443974 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:54:57.450414 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:54:57.450626 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:54:57.450638 3798194 disk_worker.go:723] disk space is not compliant W1122 07:54:57.450641 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:54:57.459017 3798194 disk_worker.go:509] system disk:vda1 I1122 07:54:57.459057 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:54:58.446008 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:54:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:54:59.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:54:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:54:59.471646 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:55:03.096145 3798194 disk_info.go:125] begin check local disk info of client I1122 07:55:03.099966 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:55:03.099975 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b4300 0xc0003b4340] E1122 07:55:06.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:55:06.412265 3798194 memory.go:184] no items to output this cycle I1122 07:55:06.412412 3798194 cpu.go:275] no items to output this cycle E1122 07:55:16.412329 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:55:16.412352 3798194 memory.go:184] no items to output this cycle I1122 07:55:16.412499 3798194 cpu.go:275] no items to output this cycle E1122 07:55:26.412489 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:55:26.412526 3798194 memory.go:191] Add success. I1122 07:55:26.413745 3798194 cpu.go:282] Add success. I1122 07:55:26.441575 3798194 net.go:642] Add success. I1122 07:55:26.443078 3798194 net.go:778] primary dev: ETH0 I1122 07:55:26.443095 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:55:26.443108 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:55:29.450008 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:55:29.450085 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:55:29.450128 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:55:36.413224 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:55:36.413251 3798194 memory.go:184] no items to output this cycle I1122 07:55:36.413398 3798194 cpu.go:275] no items to output this cycle E1122 07:55:46.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:55:46.412253 3798194 memory.go:184] no items to output this cycle I1122 07:55:46.412403 3798194 cpu.go:275] no items to output this cycle E1122 07:55:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:55:56.411945 3798194 memory.go:191] Add success. W1122 07:55:56.411973 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:55:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:55:56.411987 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:55:56.412345 3798194 cpu.go:282] Add success. I1122 07:55:56.441546 3798194 event_worker.go:152] Polling the log file for events... I1122 07:55:56.444695 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:55:56.445194 3798194 net.go:642] Add success. I1122 07:55:56.449490 3798194 net.go:778] primary dev: ETH0 I1122 07:55:56.449504 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:55:56.449519 3798194 net.go:659] Add eni metrics success, count: 7 E1122 07:55:57.450578 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 07:55:57.450592 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:55:57.450784 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:55:57.450793 3798194 disk_worker.go:723] disk space is not compliant W1122 07:55:57.450796 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:55:57.459157 3798194 disk_worker.go:509] system disk:vda1 I1122 07:55:57.459197 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 07:55:58.447803 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 07:55:58.447818 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 07:55:59.448504 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 07:55:59.450633 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:55:59.450685 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:55:59.450705 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:55:59.471226 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:56:03.103223 3798194 disk_info.go:125] begin check local disk info of client I1122 07:56:03.107041 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:56:03.107050 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002fd580 0xc0002fd5c0] E1122 07:56:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:56:06.412257 3798194 memory.go:184] no items to output this cycle I1122 07:56:06.412405 3798194 cpu.go:275] no items to output this cycle E1122 07:56:16.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:56:16.413257 3798194 memory.go:184] no items to output this cycle I1122 07:56:16.413402 3798194 cpu.go:275] no items to output this cycle E1122 07:56:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:56:26.411950 3798194 memory.go:191] Add success. I1122 07:56:26.412469 3798194 cpu.go:282] Add success. I1122 07:56:26.441305 3798194 net.go:642] Add success. I1122 07:56:26.445273 3798194 net.go:778] primary dev: ETH0 I1122 07:56:26.445288 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:56:26.445300 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:56:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:56:29.450098 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:56:29.450139 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:56:36.411952 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:56:36.411975 3798194 memory.go:184] no items to output this cycle I1122 07:56:36.412127 3798194 cpu.go:275] no items to output this cycle E1122 07:56:46.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:56:46.412263 3798194 memory.go:184] no items to output this cycle I1122 07:56:46.412415 3798194 cpu.go:275] no items to output this cycle E1122 07:56:56.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:56:56.413273 3798194 memory.go:191] Add success. W1122 07:56:56.413299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:56:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:56:56.413314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:56:56.417994 3798194 cpu.go:282] Add success. I1122 07:56:56.441358 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:56:56.441842 3798194 net.go:642] Add success. I1122 07:56:56.443336 3798194 net.go:778] primary dev: ETH0 I1122 07:56:56.443351 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:56:56.443364 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:56:57.447605 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:56:57.447855 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:56:57.447866 3798194 disk_worker.go:723] disk space is not compliant W1122 07:56:57.447870 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:56:57.451646 3798194 disk_worker.go:509] system disk:vda1 I1122 07:56:57.451686 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:56:58.446384 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:56:59.450484 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:56:59.450562 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:56:59.450588 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:56:59.471092 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:57:03.110146 3798194 disk_info.go:125] begin check local disk info of client I1122 07:57:03.113995 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:57:03.114004 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000310880 0xc0003108c0] E1122 07:57:06.412717 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:57:06.412739 3798194 memory.go:184] no items to output this cycle I1122 07:57:06.412883 3798194 cpu.go:275] no items to output this cycle E1122 07:57:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:57:16.412261 3798194 memory.go:184] no items to output this cycle I1122 07:57:16.412406 3798194 cpu.go:275] no items to output this cycle I1122 07:57:17.660400 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 07:57:17.660408 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 07:57:26.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:57:26.413271 3798194 memory.go:191] Add success. I1122 07:57:26.417403 3798194 cpu.go:282] Add success. I1122 07:57:26.442063 3798194 net.go:642] Add success. I1122 07:57:26.443747 3798194 net.go:778] primary dev: ETH0 I1122 07:57:26.443765 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:57:26.443781 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:57:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:57:29.450481 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:57:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:57:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:57:36.412268 3798194 memory.go:184] no items to output this cycle I1122 07:57:36.412413 3798194 cpu.go:275] no items to output this cycle E1122 07:57:46.413258 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:57:46.413281 3798194 memory.go:184] no items to output this cycle I1122 07:57:46.413428 3798194 cpu.go:275] no items to output this cycle E1122 07:57:56.411920 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:57:56.411955 3798194 memory.go:191] Add success. W1122 07:57:56.411984 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:57:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:57:56.411999 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:57:56.412354 3798194 cpu.go:282] Add success. I1122 07:57:56.441818 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:57:56.442371 3798194 net.go:642] Add success. I1122 07:57:56.446628 3798194 net.go:778] primary dev: ETH0 I1122 07:57:56.446644 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:57:56.446657 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:57:57.450564 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:57:57.450718 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:57:57.450768 3798194 disk_worker.go:723] disk space is not compliant W1122 07:57:57.450772 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:57:57.459178 3798194 disk_worker.go:509] system disk:vda1 I1122 07:57:57.459221 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:57:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:57:59.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:57:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:57:59.450142 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:57:59.471668 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:58:03.114105 3798194 disk_info.go:125] begin check local disk info of client I1122 07:58:03.117945 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:58:03.117953 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003ead00 0xc0003ead40] E1122 07:58:06.412098 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:58:06.412140 3798194 memory.go:184] no items to output this cycle I1122 07:58:06.412297 3798194 cpu.go:275] no items to output this cycle E1122 07:58:16.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:58:16.412252 3798194 memory.go:184] no items to output this cycle I1122 07:58:16.412396 3798194 cpu.go:275] no items to output this cycle E1122 07:58:26.411921 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:58:26.411954 3798194 memory.go:191] Add success. I1122 07:58:26.412317 3798194 cpu.go:282] Add success. I1122 07:58:26.441924 3798194 net.go:642] Add success. I1122 07:58:26.446335 3798194 net.go:778] primary dev: ETH0 I1122 07:58:26.446351 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:58:26.446365 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:58:29.450024 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:58:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:58:29.450146 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:58:36.412770 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:58:36.412793 3798194 memory.go:184] no items to output this cycle I1122 07:58:36.412947 3798194 cpu.go:275] no items to output this cycle E1122 07:58:46.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:58:46.413258 3798194 memory.go:184] no items to output this cycle I1122 07:58:46.413418 3798194 cpu.go:275] no items to output this cycle E1122 07:58:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:58:56.412264 3798194 memory.go:191] Add success. W1122 07:58:56.412295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:58:56.412306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:58:56.412310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:58:56.416541 3798194 cpu.go:282] Add success. I1122 07:58:56.441069 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:58:56.441992 3798194 net.go:642] Add success. I1122 07:58:56.444166 3798194 net.go:778] primary dev: ETH0 I1122 07:58:56.444181 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:58:56.444193 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:58:57.449628 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:58:57.449868 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:58:57.449880 3798194 disk_worker.go:723] disk space is not compliant W1122 07:58:57.449883 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:58:57.451719 3798194 disk_worker.go:509] system disk:vda1 I1122 07:58:57.451759 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:58:58.446387 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:58:59.450403 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:58:59.450479 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:58:59.450504 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:58:59.471018 3798194 disk_local_worker.go:436] Get disk info: [] I1122 07:59:03.120145 3798194 disk_info.go:125] begin check local disk info of client I1122 07:59:03.123958 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 07:59:03.123968 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004399c0 0xc000439a00] E1122 07:59:06.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:59:06.413255 3798194 memory.go:184] no items to output this cycle I1122 07:59:06.413404 3798194 cpu.go:275] no items to output this cycle E1122 07:59:16.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:59:16.413262 3798194 memory.go:184] no items to output this cycle I1122 07:59:16.413418 3798194 cpu.go:275] no items to output this cycle E1122 07:59:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:59:26.413266 3798194 memory.go:191] Add success. I1122 07:59:26.417525 3798194 cpu.go:282] Add success. I1122 07:59:26.442194 3798194 net.go:642] Add success. I1122 07:59:26.443665 3798194 net.go:778] primary dev: ETH0 I1122 07:59:26.443680 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:59:26.443692 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:59:29.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:59:29.450476 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:59:29.450502 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 07:59:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:59:36.413259 3798194 memory.go:184] no items to output this cycle I1122 07:59:36.413400 3798194 cpu.go:275] no items to output this cycle E1122 07:59:46.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:59:46.413260 3798194 memory.go:184] no items to output this cycle I1122 07:59:46.413405 3798194 cpu.go:275] no items to output this cycle E1122 07:59:56.412777 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 07:59:56.412809 3798194 memory.go:191] Add success. W1122 07:59:56.412846 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 07:59:56.412858 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 07:59:56.412861 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 07:59:56.416987 3798194 cpu.go:282] Add success. I1122 07:59:56.441402 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 07:59:56.441993 3798194 net.go:642] Add success. I1122 07:59:56.443453 3798194 net.go:778] primary dev: ETH0 I1122 07:59:56.443468 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 07:59:56.443482 3798194 net.go:659] Add eni metrics success, count: 7 I1122 07:59:57.451013 3798194 custom_config.go:106] query custom config with name: gpu W1122 07:59:57.451242 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 07:59:57.451256 3798194 disk_worker.go:723] disk space is not compliant W1122 07:59:57.451259 3798194 disk_worker.go:743] disk inode is not compliant I1122 07:59:57.459991 3798194 disk_worker.go:509] system disk:vda1 I1122 07:59:57.460033 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 07:59:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 07:59:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 07:59:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 07:59:59.450133 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 07:59:59.471653 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:00:03.124064 3798194 disk_info.go:125] begin check local disk info of client I1122 08:00:03.127942 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:00:03.127951 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00040e940 0xc00040e980] E1122 08:00:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:00:06.412258 3798194 memory.go:184] no items to output this cycle I1122 08:00:06.412590 3798194 cpu.go:275] no items to output this cycle E1122 08:00:16.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:00:16.413254 3798194 memory.go:184] no items to output this cycle I1122 08:00:16.413401 3798194 cpu.go:275] no items to output this cycle I1122 08:00:17.660537 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 08:00:17.660546 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 08:00:26.411225 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:00:26.411261 3798194 memory.go:191] Add success. I1122 08:00:26.411612 3798194 cpu.go:282] Add success. I1122 08:00:26.442278 3798194 net.go:642] Add success. I1122 08:00:26.446816 3798194 net.go:778] primary dev: ETH0 I1122 08:00:26.446832 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:00:26.446848 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:00:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:00:29.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:00:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:00:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:00:36.413255 3798194 memory.go:184] no items to output this cycle I1122 08:00:36.413397 3798194 cpu.go:275] no items to output this cycle E1122 08:00:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:00:46.412255 3798194 memory.go:184] no items to output this cycle I1122 08:00:46.412403 3798194 cpu.go:275] no items to output this cycle E1122 08:00:56.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:00:56.413275 3798194 memory.go:191] Add success. W1122 08:00:56.413305 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:00:56.413317 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:00:56.413320 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:00:56.417924 3798194 cpu.go:282] Add success. I1122 08:00:56.442809 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:00:56.443346 3798194 net.go:642] Add success. I1122 08:00:56.446319 3798194 net.go:778] primary dev: ETH0 I1122 08:00:56.446335 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:00:56.446350 3798194 net.go:659] Add eni metrics success, count: 7 E1122 08:00:57.446521 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 08:00:57.446536 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:00:57.446796 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:00:57.446807 3798194 disk_worker.go:723] disk space is not compliant W1122 08:00:57.446810 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:00:57.449669 3798194 disk_worker.go:509] system disk:vda1 I1122 08:00:57.449710 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 08:00:58.448538 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 08:00:58.448552 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 08:00:59.448968 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 08:00:59.450052 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:00:59.450107 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:00:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:00:59.471661 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:01:03.131368 3798194 disk_info.go:125] begin check local disk info of client I1122 08:01:03.135268 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:01:03.135277 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000373100 0xc000373140] E1122 08:01:06.413256 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:01:06.413279 3798194 memory.go:184] no items to output this cycle I1122 08:01:06.413425 3798194 cpu.go:275] no items to output this cycle E1122 08:01:16.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:01:16.411977 3798194 memory.go:184] no items to output this cycle I1122 08:01:16.412134 3798194 cpu.go:275] no items to output this cycle E1122 08:01:26.411915 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:01:26.411946 3798194 memory.go:191] Add success. I1122 08:01:26.412306 3798194 cpu.go:282] Add success. I1122 08:01:26.442017 3798194 net.go:642] Add success. I1122 08:01:26.446000 3798194 net.go:778] primary dev: ETH0 I1122 08:01:26.446018 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:01:26.446032 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:01:29.450405 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:01:29.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:01:29.450511 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:01:36.413181 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:01:36.413203 3798194 memory.go:184] no items to output this cycle I1122 08:01:36.413550 3798194 cpu.go:275] no items to output this cycle E1122 08:01:46.412849 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:01:46.412870 3798194 memory.go:184] no items to output this cycle I1122 08:01:46.413019 3798194 cpu.go:275] no items to output this cycle E1122 08:01:56.413297 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:01:56.413346 3798194 memory.go:191] Add success. W1122 08:01:56.413397 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:01:56.413415 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:01:56.413421 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:01:56.414168 3798194 cpu.go:282] Add success. I1122 08:01:56.441096 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:01:56.442056 3798194 net.go:642] Add success. I1122 08:01:56.443997 3798194 net.go:778] primary dev: ETH0 I1122 08:01:56.444019 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:01:56.444039 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:01:57.446463 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:01:57.446673 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:01:57.446752 3798194 disk_worker.go:723] disk space is not compliant W1122 08:01:57.446755 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:01:57.449568 3798194 disk_worker.go:509] system disk:vda1 I1122 08:01:57.449608 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:01:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:01:59.450485 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:01:59.450566 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:01:59.450592 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:01:59.471122 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:02:03.135381 3798194 disk_info.go:125] begin check local disk info of client I1122 08:02:03.139247 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:02:03.139256 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000402e00 0xc000402e40] E1122 08:02:06.413636 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:02:06.413669 3798194 memory.go:184] no items to output this cycle I1122 08:02:06.413828 3798194 cpu.go:275] no items to output this cycle E1122 08:02:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:02:16.413257 3798194 memory.go:184] no items to output this cycle I1122 08:02:16.413408 3798194 cpu.go:275] no items to output this cycle E1122 08:02:26.412643 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:02:26.412677 3798194 memory.go:191] Add success. I1122 08:02:26.416895 3798194 cpu.go:282] Add success. I1122 08:02:26.441667 3798194 net.go:642] Add success. I1122 08:02:26.443054 3798194 net.go:778] primary dev: ETH0 I1122 08:02:26.443068 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:02:26.443082 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:02:29.457167 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:02:29.457241 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:02:29.457275 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:02:36.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:02:36.412873 3798194 memory.go:184] no items to output this cycle I1122 08:02:36.413016 3798194 cpu.go:275] no items to output this cycle E1122 08:02:46.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:02:46.411971 3798194 memory.go:184] no items to output this cycle I1122 08:02:46.412127 3798194 cpu.go:275] no items to output this cycle E1122 08:02:56.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:02:56.412890 3798194 memory.go:191] Add success. W1122 08:02:56.412921 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:02:56.412933 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:02:56.412936 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:02:56.417351 3798194 cpu.go:282] Add success. I1122 08:02:56.441878 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:02:56.442465 3798194 net.go:642] Add success. I1122 08:02:56.444021 3798194 net.go:778] primary dev: ETH0 I1122 08:02:56.444035 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:02:56.444048 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:02:57.451097 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:02:57.451362 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:02:57.451374 3798194 disk_worker.go:723] disk space is not compliant W1122 08:02:57.451377 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:02:57.460015 3798194 disk_worker.go:509] system disk:vda1 I1122 08:02:57.460056 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:02:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:02:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:02:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:02:59.450143 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:02:59.471655 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:03:03.142802 3798194 disk_info.go:125] begin check local disk info of client I1122 08:03:03.146689 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:03:03.146698 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000471d80 0xc000471dc0] E1122 08:03:06.411957 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:03:06.411980 3798194 memory.go:184] no items to output this cycle I1122 08:03:06.412138 3798194 cpu.go:275] no items to output this cycle E1122 08:03:16.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:03:16.412256 3798194 memory.go:184] no items to output this cycle I1122 08:03:16.412402 3798194 cpu.go:275] no items to output this cycle I1122 08:03:17.660712 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 08:03:17.660720 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 08:03:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:03:26.412270 3798194 memory.go:191] Add success. I1122 08:03:26.416422 3798194 cpu.go:282] Add success. I1122 08:03:26.441969 3798194 net.go:642] Add success. I1122 08:03:26.443499 3798194 net.go:778] primary dev: ETH0 I1122 08:03:26.443514 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:03:26.443527 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:03:29.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:03:29.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:03:29.450497 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:03:36.412867 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:03:36.412890 3798194 memory.go:184] no items to output this cycle I1122 08:03:36.413037 3798194 cpu.go:275] no items to output this cycle E1122 08:03:46.412555 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:03:46.412578 3798194 memory.go:184] no items to output this cycle I1122 08:03:46.412730 3798194 cpu.go:275] no items to output this cycle E1122 08:03:56.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:03:56.412894 3798194 memory.go:191] Add success. W1122 08:03:56.412929 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:03:56.412942 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:03:56.412945 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:03:56.417663 3798194 cpu.go:282] Add success. I1122 08:03:56.440964 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:03:56.441551 3798194 net.go:642] Add success. I1122 08:03:56.443078 3798194 net.go:778] primary dev: ETH0 I1122 08:03:56.443092 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:03:56.443106 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:03:57.450381 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:03:57.450588 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:03:57.450599 3798194 disk_worker.go:723] disk space is not compliant W1122 08:03:57.450602 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:03:57.458896 3798194 disk_worker.go:509] system disk:vda1 I1122 08:03:57.458936 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:03:58.446510 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:03:59.450610 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:03:59.450691 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:03:59.450716 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:03:59.471192 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:04:03.149137 3798194 disk_info.go:125] begin check local disk info of client I1122 08:04:03.153010 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:04:03.153019 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00043b040 0xc00043b080] E1122 08:04:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:04:06.411982 3798194 memory.go:184] no items to output this cycle I1122 08:04:06.412139 3798194 cpu.go:275] no items to output this cycle E1122 08:04:16.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:04:16.413252 3798194 memory.go:184] no items to output this cycle I1122 08:04:16.413586 3798194 cpu.go:275] no items to output this cycle E1122 08:04:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:04:26.411956 3798194 memory.go:191] Add success. I1122 08:04:26.412332 3798194 cpu.go:282] Add success. I1122 08:04:26.441990 3798194 net.go:642] Add success. I1122 08:04:26.446237 3798194 net.go:778] primary dev: ETH0 I1122 08:04:26.446252 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:04:26.446265 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:04:29.450026 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:04:29.450108 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:04:29.450149 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:04:36.411971 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:04:36.411995 3798194 memory.go:184] no items to output this cycle I1122 08:04:36.412154 3798194 cpu.go:275] no items to output this cycle E1122 08:04:46.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:04:46.412877 3798194 memory.go:184] no items to output this cycle I1122 08:04:46.413034 3798194 cpu.go:275] no items to output this cycle E1122 08:04:56.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:04:56.413258 3798194 memory.go:191] Add success. W1122 08:04:56.413292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:04:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:04:56.413307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:04:56.418517 3798194 cpu.go:282] Add success. I1122 08:04:56.441106 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:04:56.441659 3798194 net.go:642] Add success. I1122 08:04:56.443065 3798194 net.go:778] primary dev: ETH0 I1122 08:04:56.443080 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:04:56.443093 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:04:57.449232 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:04:57.449470 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:04:57.449483 3798194 disk_worker.go:723] disk space is not compliant W1122 08:04:57.449486 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:04:57.451296 3798194 disk_worker.go:509] system disk:vda1 I1122 08:04:57.451334 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:04:58.446390 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:04:59.450492 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:04:59.450570 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:04:59.450596 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:04:59.471104 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:05:03.153136 3798194 disk_info.go:125] begin check local disk info of client I1122 08:05:03.156960 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:05:03.156969 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002f5c40 0xc0002f5c80] E1122 08:05:06.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:05:06.412883 3798194 memory.go:184] no items to output this cycle I1122 08:05:06.413024 3798194 cpu.go:275] no items to output this cycle E1122 08:05:16.412246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:05:16.412268 3798194 memory.go:184] no items to output this cycle I1122 08:05:16.412411 3798194 cpu.go:275] no items to output this cycle E1122 08:05:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:05:26.411954 3798194 memory.go:191] Add success. I1122 08:05:26.412304 3798194 cpu.go:282] Add success. I1122 08:05:26.441866 3798194 net.go:642] Add success. I1122 08:05:26.445813 3798194 net.go:778] primary dev: ETH0 I1122 08:05:26.445828 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:05:26.445842 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:05:29.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:05:29.450123 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:05:29.450160 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:05:36.412938 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:05:36.412958 3798194 memory.go:184] no items to output this cycle I1122 08:05:36.413101 3798194 cpu.go:275] no items to output this cycle E1122 08:05:46.413351 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:05:46.413378 3798194 memory.go:184] no items to output this cycle I1122 08:05:46.413660 3798194 cpu.go:275] no items to output this cycle E1122 08:05:56.411929 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:05:56.411963 3798194 memory.go:191] Add success. W1122 08:05:56.411995 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:05:56.412007 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:05:56.412010 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:05:56.412392 3798194 cpu.go:282] Add success. I1122 08:05:56.441586 3798194 event_worker.go:152] Polling the log file for events... I1122 08:05:56.444818 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:05:56.445304 3798194 net.go:642] Add success. I1122 08:05:56.449182 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795304156, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 08:05:56.449307 3798194 net.go:778] primary dev: ETH0 I1122 08:05:56.449322 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:05:56.449334 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:05:56.472105 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 08:05:57.445602 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 08:05:57.445628 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:05:57.445891 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:05:57.445903 3798194 disk_worker.go:723] disk space is not compliant W1122 08:05:57.445906 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:05:57.450951 3798194 disk_worker.go:509] system disk:vda1 I1122 08:05:57.450992 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 08:05:58.448140 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 08:05:58.448155 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 08:05:59.448668 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 08:05:59.449734 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:05:59.449788 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:05:59.449813 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:05:59.471350 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:06:03.157072 3798194 disk_info.go:125] begin check local disk info of client I1122 08:06:03.160943 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:06:03.160952 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c4240 0xc0003c4280] E1122 08:06:06.412756 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:06:06.412778 3798194 memory.go:184] no items to output this cycle I1122 08:06:06.412936 3798194 cpu.go:275] no items to output this cycle E1122 08:06:16.413243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:06:16.413270 3798194 memory.go:184] no items to output this cycle I1122 08:06:16.413418 3798194 cpu.go:275] no items to output this cycle I1122 08:06:17.661214 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 08:06:17.661223 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 08:06:26.412061 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:06:26.412097 3798194 memory.go:191] Add success. I1122 08:06:26.416577 3798194 cpu.go:282] Add success. I1122 08:06:26.442166 3798194 net.go:642] Add success. I1122 08:06:26.443783 3798194 net.go:778] primary dev: ETH0 I1122 08:06:26.443796 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:06:26.443810 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:06:29.450554 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:06:29.450638 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:06:29.450664 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:06:36.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:06:36.412872 3798194 memory.go:184] no items to output this cycle I1122 08:06:36.413013 3798194 cpu.go:275] no items to output this cycle E1122 08:06:46.411997 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:06:46.412026 3798194 memory.go:184] no items to output this cycle I1122 08:06:46.412188 3798194 cpu.go:275] no items to output this cycle E1122 08:06:56.411925 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:06:56.411958 3798194 memory.go:191] Add success. W1122 08:06:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:06:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:06:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:06:56.412357 3798194 cpu.go:282] Add success. I1122 08:06:56.441801 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:06:56.442323 3798194 net.go:642] Add success. I1122 08:06:56.446194 3798194 net.go:778] primary dev: ETH0 I1122 08:06:56.446208 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:06:56.446221 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:06:57.450776 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:06:57.451188 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:06:57.451199 3798194 disk_worker.go:723] disk space is not compliant W1122 08:06:57.451203 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:06:57.460106 3798194 disk_worker.go:509] system disk:vda1 I1122 08:06:57.460161 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:06:58.446013 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:06:59.450093 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:06:59.450193 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:06:59.450219 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:06:59.470727 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:07:03.164477 3798194 disk_info.go:125] begin check local disk info of client I1122 08:07:03.168290 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:07:03.168299 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004c1800 0xc0004c1840] E1122 08:07:06.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:07:06.412255 3798194 memory.go:184] no items to output this cycle I1122 08:07:06.412409 3798194 cpu.go:275] no items to output this cycle E1122 08:07:16.412058 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:07:16.412081 3798194 memory.go:184] no items to output this cycle I1122 08:07:16.412244 3798194 cpu.go:275] no items to output this cycle E1122 08:07:26.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:07:26.413276 3798194 memory.go:191] Add success. I1122 08:07:26.417541 3798194 cpu.go:282] Add success. I1122 08:07:26.442090 3798194 net.go:642] Add success. I1122 08:07:26.443738 3798194 net.go:778] primary dev: ETH0 I1122 08:07:26.443753 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:07:26.443766 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:07:29.450455 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:07:29.450534 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:07:29.450560 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:07:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:07:36.412264 3798194 memory.go:184] no items to output this cycle I1122 08:07:36.412409 3798194 cpu.go:275] no items to output this cycle E1122 08:07:46.411698 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:07:46.411721 3798194 memory.go:184] no items to output this cycle I1122 08:07:46.411879 3798194 cpu.go:275] no items to output this cycle E1122 08:07:56.412853 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:07:56.412885 3798194 memory.go:191] Add success. W1122 08:07:56.412918 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:07:56.412929 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:07:56.412933 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:07:56.417020 3798194 cpu.go:282] Add success. I1122 08:07:56.441417 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:07:56.441906 3798194 net.go:642] Add success. I1122 08:07:56.443422 3798194 net.go:778] primary dev: ETH0 I1122 08:07:56.443437 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:07:56.443449 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:07:57.450996 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:07:57.451186 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:07:57.451253 3798194 disk_worker.go:723] disk space is not compliant W1122 08:07:57.451256 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:07:57.459937 3798194 disk_worker.go:509] system disk:vda1 I1122 08:07:57.459982 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:07:58.446490 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:07:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:07:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:07:59.450146 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:07:59.471671 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:08:03.171184 3798194 disk_info.go:125] begin check local disk info of client I1122 08:08:03.175149 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:08:03.175158 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003eed00 0xc0003eed40] E1122 08:08:06.413208 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:08:06.413234 3798194 memory.go:184] no items to output this cycle I1122 08:08:06.413382 3798194 cpu.go:275] no items to output this cycle E1122 08:08:16.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:08:16.412253 3798194 memory.go:184] no items to output this cycle I1122 08:08:16.412397 3798194 cpu.go:275] no items to output this cycle E1122 08:08:26.411556 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:08:26.411595 3798194 memory.go:191] Add success. I1122 08:08:26.415916 3798194 cpu.go:282] Add success. I1122 08:08:26.441515 3798194 net.go:642] Add success. I1122 08:08:26.443075 3798194 net.go:778] primary dev: ETH0 I1122 08:08:26.443088 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:08:26.443101 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:08:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:08:29.450101 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:08:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:08:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:08:36.413262 3798194 memory.go:184] no items to output this cycle I1122 08:08:36.413403 3798194 cpu.go:275] no items to output this cycle E1122 08:08:46.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:08:46.413251 3798194 memory.go:184] no items to output this cycle I1122 08:08:46.413406 3798194 cpu.go:275] no items to output this cycle E1122 08:08:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:08:56.413263 3798194 memory.go:191] Add success. W1122 08:08:56.413296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:08:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:08:56.413312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:08:56.417878 3798194 cpu.go:282] Add success. I1122 08:08:56.441266 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:08:56.441784 3798194 net.go:642] Add success. I1122 08:08:56.443375 3798194 net.go:778] primary dev: ETH0 I1122 08:08:56.443389 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:08:56.443401 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:08:57.450834 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:08:57.451050 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:08:57.451061 3798194 disk_worker.go:723] disk space is not compliant W1122 08:08:57.451065 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:08:57.459898 3798194 disk_worker.go:509] system disk:vda1 I1122 08:08:57.459938 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:08:58.446554 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:08:59.450390 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:08:59.450469 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:08:59.450498 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:08:59.471065 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:09:03.178979 3798194 disk_info.go:125] begin check local disk info of client I1122 08:09:03.182887 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:09:03.182896 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000307400 0xc000307440] E1122 08:09:06.412596 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:09:06.412627 3798194 memory.go:184] no items to output this cycle I1122 08:09:06.412773 3798194 cpu.go:275] no items to output this cycle E1122 08:09:16.412210 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:09:16.412232 3798194 memory.go:184] no items to output this cycle I1122 08:09:16.412387 3798194 cpu.go:275] no items to output this cycle I1122 08:09:17.662209 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 08:09:17.662219 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 08:09:26.411375 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:09:26.411408 3798194 memory.go:191] Add success. I1122 08:09:26.411742 3798194 cpu.go:282] Add success. I1122 08:09:26.441305 3798194 net.go:642] Add success. I1122 08:09:26.445472 3798194 net.go:778] primary dev: ETH0 I1122 08:09:26.445487 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:09:26.445501 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:09:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:09:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:09:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:09:36.412410 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:09:36.412432 3798194 memory.go:184] no items to output this cycle I1122 08:09:36.412573 3798194 cpu.go:275] no items to output this cycle E1122 08:09:46.412863 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:09:46.412887 3798194 memory.go:184] no items to output this cycle I1122 08:09:46.413178 3798194 cpu.go:275] no items to output this cycle E1122 08:09:56.412856 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:09:56.412890 3798194 memory.go:191] Add success. W1122 08:09:56.412925 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:09:56.412937 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:09:56.412940 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:09:56.417228 3798194 cpu.go:282] Add success. I1122 08:09:56.441592 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:09:56.442054 3798194 net.go:642] Add success. I1122 08:09:56.443513 3798194 net.go:778] primary dev: ETH0 I1122 08:09:56.443527 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:09:56.443540 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:09:57.451059 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:09:57.451292 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:09:57.451303 3798194 disk_worker.go:723] disk space is not compliant W1122 08:09:57.451306 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:09:57.459898 3798194 disk_worker.go:509] system disk:vda1 I1122 08:09:57.459942 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:09:58.446004 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:09:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:09:59.450477 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:09:59.450504 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:09:59.471028 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:10:03.185144 3798194 disk_info.go:125] begin check local disk info of client I1122 08:10:03.189063 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:10:03.189073 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e2d80 0xc0002e2dc0] E1122 08:10:06.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:10:06.411982 3798194 memory.go:184] no items to output this cycle I1122 08:10:06.412138 3798194 cpu.go:275] no items to output this cycle E1122 08:10:16.411972 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:10:16.411998 3798194 memory.go:184] no items to output this cycle I1122 08:10:16.412164 3798194 cpu.go:275] no items to output this cycle E1122 08:10:26.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:10:26.412265 3798194 memory.go:191] Add success. I1122 08:10:26.416485 3798194 cpu.go:282] Add success. I1122 08:10:26.442034 3798194 net.go:642] Add success. I1122 08:10:26.443636 3798194 net.go:778] primary dev: ETH0 I1122 08:10:26.443658 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:10:26.443670 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:10:29.450526 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:10:29.450614 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:10:29.450640 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:10:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:10:36.412256 3798194 memory.go:184] no items to output this cycle I1122 08:10:36.412396 3798194 cpu.go:275] no items to output this cycle E1122 08:10:46.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:10:46.413259 3798194 memory.go:184] no items to output this cycle I1122 08:10:46.413402 3798194 cpu.go:275] no items to output this cycle E1122 08:10:56.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:10:56.412264 3798194 memory.go:191] Add success. W1122 08:10:56.412293 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:10:56.412304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:10:56.412307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:10:56.416962 3798194 cpu.go:282] Add success. I1122 08:10:56.444991 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:10:56.445551 3798194 net.go:642] Add success. I1122 08:10:56.447040 3798194 net.go:778] primary dev: ETH0 I1122 08:10:56.447056 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:10:56.447068 3798194 net.go:659] Add eni metrics success, count: 7 E1122 08:10:57.450918 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 08:10:57.450932 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:10:57.451143 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:10:57.451154 3798194 disk_worker.go:723] disk space is not compliant W1122 08:10:57.451157 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:10:57.459902 3798194 disk_worker.go:509] system disk:vda1 I1122 08:10:57.459947 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 08:10:58.448507 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 08:10:58.448521 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 08:10:59.448961 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 08:10:59.450035 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:10:59.450087 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:10:59.450107 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:10:59.471624 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:11:03.192146 3798194 disk_info.go:125] begin check local disk info of client I1122 08:11:03.195962 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:11:03.195971 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046e680 0xc00046e6c0] E1122 08:11:06.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:11:06.412257 3798194 memory.go:184] no items to output this cycle I1122 08:11:06.412407 3798194 cpu.go:275] no items to output this cycle E1122 08:11:16.413242 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:11:16.413265 3798194 memory.go:184] no items to output this cycle I1122 08:11:16.413418 3798194 cpu.go:275] no items to output this cycle E1122 08:11:26.412872 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:11:26.412905 3798194 memory.go:191] Add success. I1122 08:11:26.417090 3798194 cpu.go:282] Add success. I1122 08:11:26.441628 3798194 net.go:642] Add success. I1122 08:11:26.443180 3798194 net.go:778] primary dev: ETH0 I1122 08:11:26.443195 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:11:26.443209 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:11:29.450003 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:11:29.450086 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:11:29.450127 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:11:36.412851 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:11:36.412875 3798194 memory.go:184] no items to output this cycle I1122 08:11:36.413018 3798194 cpu.go:275] no items to output this cycle E1122 08:11:46.411939 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:11:46.411960 3798194 memory.go:184] no items to output this cycle I1122 08:11:46.412105 3798194 cpu.go:275] no items to output this cycle E1122 08:11:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:11:56.412266 3798194 memory.go:191] Add success. W1122 08:11:56.412298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:11:56.412310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:11:56.412313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:11:56.416395 3798194 cpu.go:282] Add success. I1122 08:11:56.441768 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:11:56.442340 3798194 net.go:642] Add success. I1122 08:11:56.443895 3798194 net.go:778] primary dev: ETH0 I1122 08:11:56.443911 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:11:56.443924 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:11:57.451000 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:11:57.451228 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:11:57.451239 3798194 disk_worker.go:723] disk space is not compliant W1122 08:11:57.451242 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:11:57.460086 3798194 disk_worker.go:509] system disk:vda1 I1122 08:11:57.460146 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:11:58.446381 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:11:59.450393 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:11:59.450473 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:11:59.450498 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:11:59.471032 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:12:03.199135 3798194 disk_info.go:125] begin check local disk info of client I1122 08:12:03.202991 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:12:03.202999 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037f8c0 0xc00037f900] E1122 08:12:06.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:12:06.411984 3798194 memory.go:184] no items to output this cycle I1122 08:12:06.412141 3798194 cpu.go:275] no items to output this cycle E1122 08:12:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:12:16.411980 3798194 memory.go:184] no items to output this cycle I1122 08:12:16.412141 3798194 cpu.go:275] no items to output this cycle I1122 08:12:17.662376 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 08:12:17.662385 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 08:12:26.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:12:26.413291 3798194 memory.go:191] Add success. I1122 08:12:26.417383 3798194 cpu.go:282] Add success. I1122 08:12:26.441927 3798194 net.go:642] Add success. I1122 08:12:26.443490 3798194 net.go:778] primary dev: ETH0 I1122 08:12:26.443504 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:12:26.443519 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:12:29.450151 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:12:29.450213 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:12:29.450242 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:12:36.412840 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:12:36.412864 3798194 memory.go:184] no items to output this cycle I1122 08:12:36.413008 3798194 cpu.go:275] no items to output this cycle E1122 08:12:46.411949 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:12:46.411971 3798194 memory.go:184] no items to output this cycle I1122 08:12:46.412132 3798194 cpu.go:275] no items to output this cycle E1122 08:12:56.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:12:56.413265 3798194 memory.go:191] Add success. W1122 08:12:56.413295 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:12:56.413306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:12:56.413310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:12:56.417394 3798194 cpu.go:282] Add success. I1122 08:12:56.441786 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:12:56.442362 3798194 net.go:642] Add success. I1122 08:12:56.443898 3798194 net.go:778] primary dev: ETH0 I1122 08:12:56.443912 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:12:56.443924 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:12:57.451070 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:12:57.451256 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:12:57.451309 3798194 disk_worker.go:723] disk space is not compliant W1122 08:12:57.451312 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:12:57.460156 3798194 disk_worker.go:509] system disk:vda1 I1122 08:12:57.460199 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:12:58.446008 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:12:59.450023 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:12:59.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:12:59.450155 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:12:59.471658 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:13:03.205472 3798194 disk_info.go:125] begin check local disk info of client I1122 08:13:03.209381 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:13:03.209389 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0002e8380 0xc0002e83c0] E1122 08:13:06.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:13:06.413265 3798194 memory.go:184] no items to output this cycle I1122 08:13:06.413408 3798194 cpu.go:275] no items to output this cycle E1122 08:13:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:13:16.411978 3798194 memory.go:184] no items to output this cycle I1122 08:13:16.412133 3798194 cpu.go:275] no items to output this cycle E1122 08:13:26.411923 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:13:26.411954 3798194 memory.go:191] Add success. I1122 08:13:26.412298 3798194 cpu.go:282] Add success. I1122 08:13:26.441755 3798194 net.go:642] Add success. I1122 08:13:26.445908 3798194 net.go:778] primary dev: ETH0 I1122 08:13:26.445923 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:13:26.445937 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:13:29.450015 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:13:29.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:13:29.450137 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:13:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:13:36.412253 3798194 memory.go:184] no items to output this cycle I1122 08:13:36.412395 3798194 cpu.go:275] no items to output this cycle E1122 08:13:46.412842 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:13:46.412865 3798194 memory.go:184] no items to output this cycle I1122 08:13:46.413014 3798194 cpu.go:275] no items to output this cycle E1122 08:13:56.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:13:56.412266 3798194 memory.go:191] Add success. W1122 08:13:56.412298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:13:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:13:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:13:56.416769 3798194 cpu.go:282] Add success. I1122 08:13:56.441292 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:13:56.441760 3798194 net.go:642] Add success. I1122 08:13:56.443474 3798194 net.go:778] primary dev: ETH0 I1122 08:13:56.443490 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:13:56.443509 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:13:57.451099 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:13:57.451282 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:13:57.451339 3798194 disk_worker.go:723] disk space is not compliant W1122 08:13:57.451342 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:13:57.459707 3798194 disk_worker.go:509] system disk:vda1 I1122 08:13:57.459750 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:13:58.446010 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:13:59.450030 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:13:59.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:13:59.450141 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:13:59.471677 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:14:03.209501 3798194 disk_info.go:125] begin check local disk info of client I1122 08:14:03.213444 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:14:03.213453 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004c0f40 0xc0004c0f80] E1122 08:14:06.413254 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:14:06.413276 3798194 memory.go:184] no items to output this cycle I1122 08:14:06.413432 3798194 cpu.go:275] no items to output this cycle E1122 08:14:16.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:14:16.413264 3798194 memory.go:184] no items to output this cycle I1122 08:14:16.413411 3798194 cpu.go:275] no items to output this cycle E1122 08:14:26.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:14:26.413273 3798194 memory.go:191] Add success. I1122 08:14:26.417567 3798194 cpu.go:282] Add success. I1122 08:14:26.442140 3798194 net.go:642] Add success. I1122 08:14:26.443612 3798194 net.go:778] primary dev: ETH0 I1122 08:14:26.443629 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:14:26.443642 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:14:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:14:29.450488 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:14:29.450515 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:14:36.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:14:36.413265 3798194 memory.go:184] no items to output this cycle I1122 08:14:36.413407 3798194 cpu.go:275] no items to output this cycle E1122 08:14:46.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:14:46.413256 3798194 memory.go:184] no items to output this cycle I1122 08:14:46.413400 3798194 cpu.go:275] no items to output this cycle E1122 08:14:56.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:14:56.413270 3798194 memory.go:191] Add success. W1122 08:14:56.413301 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:14:56.413314 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:14:56.413326 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:14:56.417907 3798194 cpu.go:282] Add success. I1122 08:14:56.441300 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:14:56.441816 3798194 net.go:642] Add success. I1122 08:14:56.443304 3798194 net.go:778] primary dev: ETH0 I1122 08:14:56.443332 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:14:56.443345 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:14:57.446422 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:14:57.446629 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:14:57.446699 3798194 disk_worker.go:723] disk space is not compliant W1122 08:14:57.446702 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:14:57.449476 3798194 disk_worker.go:509] system disk:vda1 I1122 08:14:57.449515 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:14:58.446376 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:14:59.450480 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:14:59.450558 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:14:59.450584 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:14:59.471076 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:15:03.217159 3798194 disk_info.go:125] begin check local disk info of client I1122 08:15:03.220948 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:15:03.220960 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c5f40 0xc000426000] E1122 08:15:06.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:15:06.412877 3798194 memory.go:184] no items to output this cycle I1122 08:15:06.413017 3798194 cpu.go:275] no items to output this cycle E1122 08:15:16.413252 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:15:16.413275 3798194 memory.go:184] no items to output this cycle I1122 08:15:16.413542 3798194 cpu.go:275] no items to output this cycle I1122 08:15:17.662546 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 08:15:17.662554 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 08:15:26.413388 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:15:26.413422 3798194 memory.go:191] Add success. I1122 08:15:26.417963 3798194 cpu.go:282] Add success. I1122 08:15:26.441525 3798194 net.go:642] Add success. I1122 08:15:26.443095 3798194 net.go:778] primary dev: ETH0 I1122 08:15:26.443120 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:15:26.443135 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:15:29.450385 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:15:29.450463 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:15:29.450491 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:15:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:15:36.412257 3798194 memory.go:184] no items to output this cycle I1122 08:15:36.412400 3798194 cpu.go:275] no items to output this cycle E1122 08:15:46.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:15:46.413252 3798194 memory.go:184] no items to output this cycle I1122 08:15:46.413409 3798194 cpu.go:275] no items to output this cycle E1122 08:15:56.413095 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:15:56.413142 3798194 memory.go:191] Add success. W1122 08:15:56.413178 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:15:56.413189 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:15:56.413193 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:15:56.417489 3798194 cpu.go:282] Add success. I1122 08:15:56.441656 3798194 event_worker.go:152] Polling the log file for events... I1122 08:15:56.444863 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:15:56.445405 3798194 net.go:642] Add success. I1122 08:15:56.446874 3798194 net.go:778] primary dev: ETH0 I1122 08:15:56.446890 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:15:56.446903 3798194 net.go:659] Add eni metrics success, count: 7 E1122 08:15:57.451088 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 08:15:57.451102 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:15:57.451336 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:15:57.451345 3798194 disk_worker.go:723] disk space is not compliant W1122 08:15:57.451348 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:15:57.459735 3798194 disk_worker.go:509] system disk:vda1 I1122 08:15:57.459774 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 08:15:58.448748 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 08:15:58.448760 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 08:15:59.448980 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 08:15:59.450055 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:15:59.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:15:59.450137 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:15:59.471649 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:16:03.223546 3798194 disk_info.go:125] begin check local disk info of client I1122 08:16:03.227475 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:16:03.227484 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000335640 0xc000335680] E1122 08:16:06.413245 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:16:06.413268 3798194 memory.go:184] no items to output this cycle I1122 08:16:06.413411 3798194 cpu.go:275] no items to output this cycle E1122 08:16:16.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:16:16.411977 3798194 memory.go:184] no items to output this cycle I1122 08:16:16.412134 3798194 cpu.go:275] no items to output this cycle E1122 08:16:26.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:16:26.411949 3798194 memory.go:191] Add success. I1122 08:16:26.412309 3798194 cpu.go:282] Add success. I1122 08:16:26.441946 3798194 net.go:642] Add success. I1122 08:16:26.445919 3798194 net.go:778] primary dev: ETH0 I1122 08:16:26.445934 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:16:26.445947 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:16:29.450019 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:16:29.450094 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:16:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:16:36.412944 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:16:36.412967 3798194 memory.go:184] no items to output this cycle I1122 08:16:36.413134 3798194 cpu.go:275] no items to output this cycle E1122 08:16:46.411960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:16:46.411982 3798194 memory.go:184] no items to output this cycle I1122 08:16:46.412272 3798194 cpu.go:275] no items to output this cycle E1122 08:16:56.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:16:56.412273 3798194 memory.go:191] Add success. W1122 08:16:56.412300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:16:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:16:56.412316 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:16:56.416438 3798194 cpu.go:282] Add success. I1122 08:16:56.441840 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:16:56.442468 3798194 net.go:642] Add success. I1122 08:16:56.443909 3798194 net.go:778] primary dev: ETH0 I1122 08:16:56.443924 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:16:56.443937 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:16:57.451293 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:16:57.451492 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:16:57.451502 3798194 disk_worker.go:723] disk space is not compliant W1122 08:16:57.451505 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:16:57.459846 3798194 disk_worker.go:509] system disk:vda1 I1122 08:16:57.459888 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:16:58.445703 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:16:59.449673 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:16:59.449748 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:16:59.449773 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:16:59.471298 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:17:03.231147 3798194 disk_info.go:125] begin check local disk info of client I1122 08:17:03.235059 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:17:03.235067 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000309040 0xc000309080] E1122 08:17:06.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:17:06.412881 3798194 memory.go:184] no items to output this cycle I1122 08:17:06.413026 3798194 cpu.go:275] no items to output this cycle E1122 08:17:16.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:17:16.413261 3798194 memory.go:184] no items to output this cycle I1122 08:17:16.413406 3798194 cpu.go:275] no items to output this cycle E1122 08:17:26.411918 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:17:26.411951 3798194 memory.go:191] Add success. I1122 08:17:26.412315 3798194 cpu.go:282] Add success. I1122 08:17:26.442075 3798194 net.go:642] Add success. I1122 08:17:26.446231 3798194 net.go:778] primary dev: ETH0 I1122 08:17:26.446247 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:17:26.446260 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:17:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:17:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:17:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:17:36.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:17:36.412254 3798194 memory.go:184] no items to output this cycle I1122 08:17:36.412394 3798194 cpu.go:275] no items to output this cycle E1122 08:17:46.412941 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:17:46.412965 3798194 memory.go:184] no items to output this cycle I1122 08:17:46.413132 3798194 cpu.go:275] no items to output this cycle E1122 08:17:56.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:17:56.413277 3798194 memory.go:191] Add success. W1122 08:17:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:17:56.413315 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:17:56.413318 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:17:56.417359 3798194 cpu.go:282] Add success. I1122 08:17:56.441864 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:17:56.442369 3798194 net.go:642] Add success. I1122 08:17:56.443818 3798194 net.go:778] primary dev: ETH0 I1122 08:17:56.443833 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:17:56.443847 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:17:57.450691 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:17:57.450845 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:17:57.450895 3798194 disk_worker.go:723] disk space is not compliant W1122 08:17:57.450898 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:17:57.459412 3798194 disk_worker.go:509] system disk:vda1 I1122 08:17:57.459476 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:17:58.445858 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:17:59.449960 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:17:59.450033 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:17:59.450060 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:17:59.471583 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:18:03.235171 3798194 disk_info.go:125] begin check local disk info of client I1122 08:18:03.239020 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:18:03.239029 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a11c0 0xc0004a1200] E1122 08:18:06.412799 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:18:06.412823 3798194 memory.go:184] no items to output this cycle I1122 08:18:06.412966 3798194 cpu.go:275] no items to output this cycle E1122 08:18:16.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:18:16.411986 3798194 memory.go:184] no items to output this cycle I1122 08:18:16.412147 3798194 cpu.go:275] no items to output this cycle I1122 08:18:17.662670 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 08:18:17.662678 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 08:18:26.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:18:26.412892 3798194 memory.go:191] Add success. I1122 08:18:26.417048 3798194 cpu.go:282] Add success. I1122 08:18:26.441614 3798194 net.go:642] Add success. I1122 08:18:26.443106 3798194 net.go:778] primary dev: ETH0 I1122 08:18:26.443136 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:18:26.443149 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:18:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:18:29.450485 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:18:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:18:36.413228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:18:36.413252 3798194 memory.go:184] no items to output this cycle I1122 08:18:36.413396 3798194 cpu.go:275] no items to output this cycle E1122 08:18:46.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:18:46.411976 3798194 memory.go:184] no items to output this cycle I1122 08:18:46.412147 3798194 cpu.go:275] no items to output this cycle E1122 08:18:56.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:18:56.413278 3798194 memory.go:191] Add success. W1122 08:18:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:18:56.413320 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:18:56.413323 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:18:56.417526 3798194 cpu.go:282] Add success. I1122 08:18:56.441951 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:18:56.442546 3798194 net.go:642] Add success. I1122 08:18:56.443909 3798194 net.go:778] primary dev: ETH0 I1122 08:18:56.443924 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:18:56.443936 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:18:57.450628 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:18:57.450803 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:18:57.450860 3798194 disk_worker.go:723] disk space is not compliant W1122 08:18:57.450863 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:18:57.459154 3798194 disk_worker.go:509] system disk:vda1 I1122 08:18:57.459195 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:18:58.446386 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:18:59.450504 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:18:59.450584 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:18:59.450612 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:18:59.471133 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:19:03.242137 3798194 disk_info.go:125] begin check local disk info of client I1122 08:19:03.245967 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:19:03.245975 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003b4280 0xc0003b42c0] E1122 08:19:06.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:19:06.411983 3798194 memory.go:184] no items to output this cycle I1122 08:19:06.412140 3798194 cpu.go:275] no items to output this cycle E1122 08:19:16.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:19:16.412254 3798194 memory.go:184] no items to output this cycle I1122 08:19:16.412403 3798194 cpu.go:275] no items to output this cycle E1122 08:19:26.412222 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:19:26.412254 3798194 memory.go:191] Add success. I1122 08:19:26.416505 3798194 cpu.go:282] Add success. I1122 08:19:26.442538 3798194 net.go:642] Add success. I1122 08:19:26.443968 3798194 net.go:778] primary dev: ETH0 I1122 08:19:26.443992 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:19:26.444017 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:19:29.450409 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:19:29.450498 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:19:29.450527 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:19:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:19:36.413261 3798194 memory.go:184] no items to output this cycle I1122 08:19:36.413409 3798194 cpu.go:275] no items to output this cycle E1122 08:19:46.412338 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:19:46.412360 3798194 memory.go:184] no items to output this cycle I1122 08:19:46.412505 3798194 cpu.go:275] no items to output this cycle E1122 08:19:56.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:19:56.413272 3798194 memory.go:191] Add success. W1122 08:19:56.413300 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:19:56.413311 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:19:56.413314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:19:56.417872 3798194 cpu.go:282] Add success. I1122 08:19:56.441241 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:19:56.441745 3798194 net.go:642] Add success. I1122 08:19:56.443172 3798194 net.go:778] primary dev: ETH0 I1122 08:19:56.443186 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:19:56.443199 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:19:57.450942 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:19:57.451174 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:19:57.451186 3798194 disk_worker.go:723] disk space is not compliant W1122 08:19:57.451189 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:19:57.460007 3798194 disk_worker.go:509] system disk:vda1 I1122 08:19:57.460051 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:19:58.445934 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:19:59.450520 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:19:59.450610 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:19:59.450636 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:19:59.471127 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:20:03.249955 3798194 disk_info.go:125] begin check local disk info of client I1122 08:20:03.253855 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:20:03.253864 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00043c6c0 0xc00043c700] E1122 08:20:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:20:06.411980 3798194 memory.go:184] no items to output this cycle I1122 08:20:06.412140 3798194 cpu.go:275] no items to output this cycle E1122 08:20:16.412525 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:20:16.412549 3798194 memory.go:184] no items to output this cycle I1122 08:20:16.412691 3798194 cpu.go:275] no items to output this cycle E1122 08:20:26.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:20:26.413272 3798194 memory.go:191] Add success. I1122 08:20:26.417401 3798194 cpu.go:282] Add success. I1122 08:20:26.441924 3798194 net.go:642] Add success. I1122 08:20:26.443274 3798194 net.go:778] primary dev: ETH0 I1122 08:20:26.443289 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:20:26.443302 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:20:29.450401 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:20:29.450484 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:20:29.450510 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:20:36.411981 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:20:36.412004 3798194 memory.go:184] no items to output this cycle I1122 08:20:36.412167 3798194 cpu.go:275] no items to output this cycle E1122 08:20:46.411947 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:20:46.411971 3798194 memory.go:184] no items to output this cycle I1122 08:20:46.412141 3798194 cpu.go:275] no items to output this cycle E1122 08:20:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:20:56.413269 3798194 memory.go:191] Add success. W1122 08:20:56.413296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:20:56.413307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:20:56.413311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:20:56.417534 3798194 cpu.go:282] Add success. I1122 08:20:56.444808 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:20:56.445853 3798194 net.go:642] Add success. I1122 08:20:56.447445 3798194 net.go:778] primary dev: ETH0 I1122 08:20:56.447465 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:20:56.447573 3798194 net.go:659] Add eni metrics success, count: 7 E1122 08:20:57.451279 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 08:20:57.451293 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:20:57.451500 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:20:57.451510 3798194 disk_worker.go:723] disk space is not compliant W1122 08:20:57.451513 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:20:57.459967 3798194 disk_worker.go:509] system disk:vda1 I1122 08:20:57.460010 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 08:20:58.448460 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 08:20:58.448476 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 08:20:59.448967 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 08:20:59.450042 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:20:59.450093 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:20:59.450122 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:20:59.471668 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:21:03.253972 3798194 disk_info.go:125] begin check local disk info of client I1122 08:21:03.257757 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:21:03.257766 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004c2400 0xc0004c2440] E1122 08:21:06.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:21:06.413256 3798194 memory.go:184] no items to output this cycle I1122 08:21:06.413400 3798194 cpu.go:275] no items to output this cycle E1122 08:21:16.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:21:16.411982 3798194 memory.go:184] no items to output this cycle I1122 08:21:16.412149 3798194 cpu.go:275] no items to output this cycle I1122 08:21:17.662852 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 08:21:17.662860 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 08:21:26.413350 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:21:26.413382 3798194 memory.go:191] Add success. I1122 08:21:26.418013 3798194 cpu.go:282] Add success. I1122 08:21:26.441572 3798194 net.go:642] Add success. I1122 08:21:26.443131 3798194 net.go:778] primary dev: ETH0 I1122 08:21:26.443145 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:21:26.443158 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:21:29.450206 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:21:29.450283 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:21:29.450311 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:21:36.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:21:36.413257 3798194 memory.go:184] no items to output this cycle I1122 08:21:36.413394 3798194 cpu.go:275] no items to output this cycle E1122 08:21:46.411953 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:21:46.411975 3798194 memory.go:184] no items to output this cycle I1122 08:21:46.412127 3798194 cpu.go:275] no items to output this cycle E1122 08:21:56.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:21:56.411959 3798194 memory.go:191] Add success. W1122 08:21:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:21:56.412000 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:21:56.412003 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:21:56.412342 3798194 cpu.go:282] Add success. I1122 08:21:56.441771 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:21:56.442378 3798194 net.go:642] Add success. I1122 08:21:56.448263 3798194 net.go:778] primary dev: ETH0 I1122 08:21:56.448277 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:21:56.448290 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:21:57.450633 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:21:57.450797 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:21:57.450850 3798194 disk_worker.go:723] disk space is not compliant W1122 08:21:57.450865 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:21:57.459216 3798194 disk_worker.go:509] system disk:vda1 I1122 08:21:57.459257 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:21:58.446420 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:21:59.449956 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:21:59.450039 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:21:59.450066 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:21:59.471622 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:22:03.257861 3798194 disk_info.go:125] begin check local disk info of client I1122 08:22:03.261708 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:22:03.261716 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000410a80 0xc000410ac0] E1122 08:22:06.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:22:06.413259 3798194 memory.go:184] no items to output this cycle I1122 08:22:06.413533 3798194 cpu.go:275] no items to output this cycle E1122 08:22:16.413246 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:22:16.413270 3798194 memory.go:184] no items to output this cycle I1122 08:22:16.413420 3798194 cpu.go:275] no items to output this cycle E1122 08:22:26.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:22:26.412279 3798194 memory.go:191] Add success. I1122 08:22:26.417073 3798194 cpu.go:282] Add success. I1122 08:22:26.441648 3798194 net.go:642] Add success. I1122 08:22:26.443060 3798194 net.go:778] primary dev: ETH0 I1122 08:22:26.443074 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:22:26.443088 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:22:29.450014 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:22:29.450105 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:22:29.450147 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:22:36.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:22:36.411980 3798194 memory.go:184] no items to output this cycle I1122 08:22:36.412139 3798194 cpu.go:275] no items to output this cycle E1122 08:22:46.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:22:46.412255 3798194 memory.go:184] no items to output this cycle I1122 08:22:46.412402 3798194 cpu.go:275] no items to output this cycle E1122 08:22:56.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:22:56.412268 3798194 memory.go:191] Add success. W1122 08:22:56.412299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:22:56.412312 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:22:56.412315 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:22:56.416937 3798194 cpu.go:282] Add success. I1122 08:22:56.441323 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:22:56.441789 3798194 net.go:642] Add success. I1122 08:22:56.443324 3798194 net.go:778] primary dev: ETH0 I1122 08:22:56.443339 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:22:56.443363 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:22:57.452795 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:22:57.453008 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:22:57.453019 3798194 disk_worker.go:723] disk space is not compliant W1122 08:22:57.453022 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:22:57.464932 3798194 disk_worker.go:509] system disk:vda1 I1122 08:22:57.464983 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:22:58.445989 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:22:59.450022 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:22:59.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:22:59.450147 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:22:59.471636 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:23:03.264146 3798194 disk_info.go:125] begin check local disk info of client I1122 08:23:03.267972 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:23:03.267981 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00037cfc0 0xc00037d000] E1122 08:23:06.412229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:23:06.412253 3798194 memory.go:184] no items to output this cycle I1122 08:23:06.412393 3798194 cpu.go:275] no items to output this cycle E1122 08:23:16.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:23:16.413259 3798194 memory.go:184] no items to output this cycle I1122 08:23:16.413408 3798194 cpu.go:275] no items to output this cycle E1122 08:23:26.412857 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:23:26.412893 3798194 memory.go:191] Add success. I1122 08:23:26.417290 3798194 cpu.go:282] Add success. I1122 08:23:26.441716 3798194 net.go:642] Add success. I1122 08:23:26.443163 3798194 net.go:778] primary dev: ETH0 I1122 08:23:26.443177 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:23:26.443192 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:23:29.450399 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:23:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:23:29.450499 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:23:36.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:23:36.413254 3798194 memory.go:184] no items to output this cycle I1122 08:23:36.413397 3798194 cpu.go:275] no items to output this cycle E1122 08:23:46.412239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:23:46.412266 3798194 memory.go:184] no items to output this cycle I1122 08:23:46.412419 3798194 cpu.go:275] no items to output this cycle E1122 08:23:56.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:23:56.413270 3798194 memory.go:191] Add success. W1122 08:23:56.413299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:23:56.413310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:23:56.413313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:23:56.417601 3798194 cpu.go:282] Add success. I1122 08:23:56.440925 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:23:56.441466 3798194 net.go:642] Add success. I1122 08:23:56.442891 3798194 net.go:778] primary dev: ETH0 I1122 08:23:56.442906 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:23:56.442920 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:23:57.449173 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:23:57.449404 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:23:57.449416 3798194 disk_worker.go:723] disk space is not compliant W1122 08:23:57.449419 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:23:57.451267 3798194 disk_worker.go:509] system disk:vda1 I1122 08:23:57.451310 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:23:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:23:59.450486 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:23:59.450566 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:23:59.450592 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:23:59.471130 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:24:03.271144 3798194 disk_info.go:125] begin check local disk info of client I1122 08:24:03.275128 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:24:03.275139 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00041b3c0 0xc00041b400] E1122 08:24:06.413044 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:24:06.413066 3798194 memory.go:184] no items to output this cycle I1122 08:24:06.413237 3798194 cpu.go:275] no items to output this cycle E1122 08:24:16.413012 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:24:16.413034 3798194 memory.go:184] no items to output this cycle I1122 08:24:16.413192 3798194 cpu.go:275] no items to output this cycle I1122 08:24:17.663215 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 08:24:17.663224 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 08:24:26.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:24:26.412272 3798194 memory.go:191] Add success. I1122 08:24:26.416337 3798194 cpu.go:282] Add success. I1122 08:24:26.441874 3798194 net.go:642] Add success. I1122 08:24:26.443338 3798194 net.go:778] primary dev: ETH0 I1122 08:24:26.443353 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:24:26.443367 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:24:29.450133 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:24:29.450210 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:24:29.450236 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:24:36.413103 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:24:36.413136 3798194 memory.go:184] no items to output this cycle I1122 08:24:36.413284 3798194 cpu.go:275] no items to output this cycle E1122 08:24:46.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:24:46.412255 3798194 memory.go:184] no items to output this cycle I1122 08:24:46.412397 3798194 cpu.go:275] no items to output this cycle E1122 08:24:56.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:24:56.413272 3798194 memory.go:191] Add success. W1122 08:24:56.413299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:24:56.413310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:24:56.413313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:24:56.417779 3798194 cpu.go:282] Add success. I1122 08:24:56.441156 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:24:56.441635 3798194 net.go:642] Add success. I1122 08:24:56.443418 3798194 net.go:778] primary dev: ETH0 I1122 08:24:56.443432 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:24:56.443444 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:24:57.451729 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:24:57.451892 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:24:57.451945 3798194 disk_worker.go:723] disk space is not compliant W1122 08:24:57.451948 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:24:57.460434 3798194 disk_worker.go:509] system disk:vda1 I1122 08:24:57.460473 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:24:58.446372 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:24:59.450476 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:24:59.450564 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:24:59.450589 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:24:59.471124 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:25:03.275245 3798194 disk_info.go:125] begin check local disk info of client I1122 08:25:03.279036 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:25:03.279045 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004898c0 0xc000489900] E1122 08:25:06.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:25:06.412252 3798194 memory.go:184] no items to output this cycle I1122 08:25:06.412397 3798194 cpu.go:275] no items to output this cycle E1122 08:25:16.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:25:16.413258 3798194 memory.go:184] no items to output this cycle I1122 08:25:16.413403 3798194 cpu.go:275] no items to output this cycle E1122 08:25:26.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:25:26.412280 3798194 memory.go:191] Add success. I1122 08:25:26.416695 3798194 cpu.go:282] Add success. I1122 08:25:26.442287 3798194 net.go:642] Add success. I1122 08:25:26.443877 3798194 net.go:778] primary dev: ETH0 I1122 08:25:26.443893 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:25:26.443908 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:25:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:25:29.450472 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:25:29.450498 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:25:36.412236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:25:36.412262 3798194 memory.go:184] no items to output this cycle I1122 08:25:36.412405 3798194 cpu.go:275] no items to output this cycle E1122 08:25:46.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:25:46.411978 3798194 memory.go:184] no items to output this cycle I1122 08:25:46.412133 3798194 cpu.go:275] no items to output this cycle E1122 08:25:56.411917 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:25:56.411948 3798194 memory.go:191] Add success. W1122 08:25:56.411974 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:25:56.411985 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:25:56.411989 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:25:56.412372 3798194 cpu.go:282] Add success. I1122 08:25:56.441692 3798194 event_worker.go:152] Polling the log file for events... I1122 08:25:56.444884 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:25:56.445419 3798194 net.go:642] Add success. I1122 08:25:56.449724 3798194 net.go:778] primary dev: ETH0 I1122 08:25:56.449740 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:25:56.449752 3798194 net.go:659] Add eni metrics success, count: 7 E1122 08:25:57.450534 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 08:25:57.450550 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:25:57.450747 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:25:57.450757 3798194 disk_worker.go:723] disk space is not compliant W1122 08:25:57.450760 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:25:57.459033 3798194 disk_worker.go:509] system disk:vda1 I1122 08:25:57.459078 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 08:25:58.448089 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 08:25:58.448102 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 08:25:59.448728 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 08:25:59.449805 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:25:59.449856 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:25:59.449876 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:25:59.471407 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:26:03.282951 3798194 disk_info.go:125] begin check local disk info of client I1122 08:26:03.286738 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:26:03.286747 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003d6ec0 0xc0003d6f00] E1122 08:26:06.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:26:06.413256 3798194 memory.go:184] no items to output this cycle I1122 08:26:06.413400 3798194 cpu.go:275] no items to output this cycle E1122 08:26:16.411963 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:26:16.411989 3798194 memory.go:184] no items to output this cycle I1122 08:26:16.412263 3798194 cpu.go:275] no items to output this cycle E1122 08:26:26.412864 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:26:26.412912 3798194 memory.go:191] Add success. I1122 08:26:26.417076 3798194 cpu.go:282] Add success. I1122 08:26:26.441615 3798194 net.go:642] Add success. I1122 08:26:26.443076 3798194 net.go:778] primary dev: ETH0 I1122 08:26:26.443090 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:26:26.443103 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:26:29.450396 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:26:29.450480 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:26:29.450509 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:26:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:26:36.412255 3798194 memory.go:184] no items to output this cycle I1122 08:26:36.412412 3798194 cpu.go:275] no items to output this cycle E1122 08:26:46.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:26:46.412256 3798194 memory.go:184] no items to output this cycle I1122 08:26:46.412401 3798194 cpu.go:275] no items to output this cycle E1122 08:26:56.411236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:26:56.411268 3798194 memory.go:191] Add success. W1122 08:26:56.411294 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:26:56.411306 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:26:56.411310 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:26:56.415509 3798194 cpu.go:282] Add success. I1122 08:26:56.441902 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:26:56.442466 3798194 net.go:642] Add success. I1122 08:26:56.443980 3798194 net.go:778] primary dev: ETH0 I1122 08:26:56.443995 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:26:56.444006 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:26:57.450962 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:26:57.451203 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:26:57.451215 3798194 disk_worker.go:723] disk space is not compliant W1122 08:26:57.451218 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:26:57.460184 3798194 disk_worker.go:509] system disk:vda1 I1122 08:26:57.460225 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:26:58.446657 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:26:59.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:26:59.450471 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:26:59.450496 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:26:59.471005 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:27:03.291174 3798194 disk_info.go:125] begin check local disk info of client I1122 08:27:03.294746 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:27:03.294755 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000315440 0xc000315480] E1122 08:27:06.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:27:06.413254 3798194 memory.go:184] no items to output this cycle I1122 08:27:06.413394 3798194 cpu.go:275] no items to output this cycle E1122 08:27:16.412238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:27:16.412259 3798194 memory.go:184] no items to output this cycle I1122 08:27:16.412404 3798194 cpu.go:275] no items to output this cycle I1122 08:27:17.664222 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 08:27:17.664230 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 08:27:26.412827 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:27:26.412861 3798194 memory.go:191] Add success. I1122 08:27:26.417175 3798194 cpu.go:282] Add success. I1122 08:27:26.441721 3798194 net.go:642] Add success. I1122 08:27:26.443336 3798194 net.go:778] primary dev: ETH0 I1122 08:27:26.443350 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:27:26.443366 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:27:29.450412 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:27:29.450495 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:27:29.450520 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:27:36.413305 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:27:36.413328 3798194 memory.go:184] no items to output this cycle I1122 08:27:36.413470 3798194 cpu.go:275] no items to output this cycle E1122 08:27:46.411941 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:27:46.411965 3798194 memory.go:184] no items to output this cycle I1122 08:27:46.412244 3798194 cpu.go:275] no items to output this cycle E1122 08:27:56.412847 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:27:56.412878 3798194 memory.go:191] Add success. W1122 08:27:56.412906 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:27:56.412917 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:27:56.412920 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:27:56.417236 3798194 cpu.go:282] Add success. I1122 08:27:56.441635 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:27:56.442183 3798194 net.go:642] Add success. I1122 08:27:56.443528 3798194 net.go:778] primary dev: ETH0 I1122 08:27:56.443542 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:27:56.443555 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:27:57.450949 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:27:57.451105 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:27:57.451178 3798194 disk_worker.go:723] disk space is not compliant W1122 08:27:57.451181 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:27:57.459982 3798194 disk_worker.go:509] system disk:vda1 I1122 08:27:57.460025 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:27:58.446003 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:27:59.450108 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:27:59.450194 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:27:59.450223 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:27:59.470706 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:28:03.298575 3798194 disk_info.go:125] begin check local disk info of client I1122 08:28:03.302414 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:28:03.302423 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000315580 0xc0003155c0] E1122 08:28:06.413501 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:28:06.413523 3798194 memory.go:184] no items to output this cycle I1122 08:28:06.413671 3798194 cpu.go:275] no items to output this cycle E1122 08:28:16.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:28:16.413265 3798194 memory.go:184] no items to output this cycle I1122 08:28:16.413421 3798194 cpu.go:275] no items to output this cycle E1122 08:28:26.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:28:26.412267 3798194 memory.go:191] Add success. I1122 08:28:26.416494 3798194 cpu.go:282] Add success. I1122 08:28:26.442067 3798194 net.go:642] Add success. I1122 08:28:26.443459 3798194 net.go:778] primary dev: ETH0 I1122 08:28:26.443473 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:28:26.443485 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:28:29.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:28:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:28:29.450133 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:28:36.413252 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:28:36.413275 3798194 memory.go:184] no items to output this cycle I1122 08:28:36.413417 3798194 cpu.go:275] no items to output this cycle E1122 08:28:46.411948 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:28:46.411970 3798194 memory.go:184] no items to output this cycle I1122 08:28:46.412126 3798194 cpu.go:275] no items to output this cycle E1122 08:28:56.412231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:28:56.412266 3798194 memory.go:191] Add success. W1122 08:28:56.412292 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:28:56.412304 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:28:56.412307 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:28:56.416854 3798194 cpu.go:282] Add success. I1122 08:28:56.441233 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:28:56.441708 3798194 net.go:642] Add success. I1122 08:28:56.443175 3798194 net.go:778] primary dev: ETH0 I1122 08:28:56.443191 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:28:56.443203 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:28:57.450382 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:28:57.450577 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:28:57.450588 3798194 disk_worker.go:723] disk space is not compliant W1122 08:28:57.450591 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:28:57.458999 3798194 disk_worker.go:509] system disk:vda1 I1122 08:28:57.459064 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:28:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:28:59.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:28:59.450474 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:28:59.450501 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:28:59.471072 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:29:03.306146 3798194 disk_info.go:125] begin check local disk info of client I1122 08:29:03.310012 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:29:03.310020 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004a9140 0xc0004a9180] E1122 08:29:06.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:29:06.412260 3798194 memory.go:184] no items to output this cycle I1122 08:29:06.412411 3798194 cpu.go:275] no items to output this cycle E1122 08:29:16.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:29:16.411978 3798194 memory.go:184] no items to output this cycle I1122 08:29:16.412139 3798194 cpu.go:275] no items to output this cycle E1122 08:29:26.413240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:29:26.413275 3798194 memory.go:191] Add success. I1122 08:29:26.417544 3798194 cpu.go:282] Add success. I1122 08:29:26.442072 3798194 net.go:642] Add success. I1122 08:29:26.443526 3798194 net.go:778] primary dev: ETH0 I1122 08:29:26.443542 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:29:26.443555 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:29:29.450020 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:29:29.450106 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:29:29.450143 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:29:36.411954 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:29:36.411978 3798194 memory.go:184] no items to output this cycle I1122 08:29:36.412135 3798194 cpu.go:275] no items to output this cycle E1122 08:29:46.413048 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:29:46.413069 3798194 memory.go:184] no items to output this cycle I1122 08:29:46.413236 3798194 cpu.go:275] no items to output this cycle E1122 08:29:56.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:29:56.412263 3798194 memory.go:191] Add success. W1122 08:29:56.412291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:29:56.412302 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:29:56.412305 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:29:56.416831 3798194 cpu.go:282] Add success. I1122 08:29:56.441258 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:29:56.441828 3798194 net.go:642] Add success. I1122 08:29:56.443443 3798194 net.go:778] primary dev: ETH0 I1122 08:29:56.443458 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:29:56.443470 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:29:57.450768 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:29:57.450962 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:29:57.450973 3798194 disk_worker.go:723] disk space is not compliant W1122 08:29:57.450977 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:29:57.459701 3798194 disk_worker.go:509] system disk:vda1 I1122 08:29:57.459747 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:29:58.446370 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:29:59.450391 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:29:59.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:29:59.450492 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:29:59.471016 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:30:03.313136 3798194 disk_info.go:125] begin check local disk info of client I1122 08:30:03.316964 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:30:03.316973 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003c8280 0xc0003c82c0] E1122 08:30:06.411048 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:30:06.411071 3798194 memory.go:184] no items to output this cycle I1122 08:30:06.411233 3798194 cpu.go:275] no items to output this cycle E1122 08:30:16.413239 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:30:16.413263 3798194 memory.go:184] no items to output this cycle I1122 08:30:16.413406 3798194 cpu.go:275] no items to output this cycle I1122 08:30:17.665210 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 08:30:17.665219 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 08:30:26.412828 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:30:26.412861 3798194 memory.go:191] Add success. I1122 08:30:26.416999 3798194 cpu.go:282] Add success. I1122 08:30:26.441879 3798194 net.go:642] Add success. I1122 08:30:26.443446 3798194 net.go:778] primary dev: ETH0 I1122 08:30:26.443461 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:30:26.443483 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:30:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:30:29.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:30:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:30:36.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:30:36.412258 3798194 memory.go:184] no items to output this cycle I1122 08:30:36.412399 3798194 cpu.go:275] no items to output this cycle E1122 08:30:46.413226 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:30:46.413248 3798194 memory.go:184] no items to output this cycle I1122 08:30:46.413396 3798194 cpu.go:275] no items to output this cycle E1122 08:30:56.413227 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:30:56.413259 3798194 memory.go:191] Add success. W1122 08:30:56.413286 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:30:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:30:56.413300 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:30:56.417731 3798194 cpu.go:282] Add success. I1122 08:30:56.443971 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:30:56.444551 3798194 net.go:642] Add success. I1122 08:30:56.445899 3798194 net.go:778] primary dev: ETH0 I1122 08:30:56.445913 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:30:56.445927 3798194 net.go:659] Add eni metrics success, count: 7 E1122 08:30:57.451060 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 08:30:57.451075 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:30:57.451245 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:30:57.451297 3798194 disk_worker.go:723] disk space is not compliant W1122 08:30:57.451300 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:30:57.460006 3798194 disk_worker.go:509] system disk:vda1 I1122 08:30:57.460045 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 08:30:58.448450 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 08:30:58.448465 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 08:30:59.448977 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 08:30:59.450052 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:30:59.450103 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:30:59.450129 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:30:59.471630 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:31:03.321145 3798194 disk_info.go:125] begin check local disk info of client I1122 08:31:03.324934 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:31:03.324942 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00025d780 0xc00025d7c0] E1122 08:31:06.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:31:06.412881 3798194 memory.go:184] no items to output this cycle I1122 08:31:06.413027 3798194 cpu.go:275] no items to output this cycle E1122 08:31:16.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:31:16.412264 3798194 memory.go:184] no items to output this cycle I1122 08:31:16.412413 3798194 cpu.go:275] no items to output this cycle E1122 08:31:26.412866 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:31:26.412900 3798194 memory.go:191] Add success. I1122 08:31:26.417284 3798194 cpu.go:282] Add success. I1122 08:31:26.441745 3798194 net.go:642] Add success. I1122 08:31:26.443269 3798194 net.go:778] primary dev: ETH0 I1122 08:31:26.443283 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:31:26.443296 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:31:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:31:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:31:29.450141 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:31:36.412833 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:31:36.412854 3798194 memory.go:184] no items to output this cycle I1122 08:31:36.412996 3798194 cpu.go:275] no items to output this cycle E1122 08:31:46.412862 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:31:46.412886 3798194 memory.go:184] no items to output this cycle I1122 08:31:46.413032 3798194 cpu.go:275] no items to output this cycle E1122 08:31:56.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:31:56.412283 3798194 memory.go:191] Add success. W1122 08:31:56.412319 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:31:56.412331 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:31:56.412334 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:31:56.416452 3798194 cpu.go:282] Add success. I1122 08:31:56.440960 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:31:56.441556 3798194 net.go:642] Add success. I1122 08:31:56.442902 3798194 net.go:778] primary dev: ETH0 I1122 08:31:56.442916 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:31:56.442929 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:31:57.451359 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:31:57.451519 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:31:57.451574 3798194 disk_worker.go:723] disk space is not compliant W1122 08:31:57.451577 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:31:57.459887 3798194 disk_worker.go:509] system disk:vda1 I1122 08:31:57.459930 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:31:58.446000 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:31:59.450013 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:31:59.450095 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:31:59.450135 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:31:59.471653 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:32:03.329352 3798194 disk_info.go:125] begin check local disk info of client I1122 08:32:03.333184 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:32:03.333192 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030eb40 0xc00030eb80] E1122 08:32:06.413785 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:32:06.413814 3798194 memory.go:184] no items to output this cycle I1122 08:32:06.413965 3798194 cpu.go:275] no items to output this cycle E1122 08:32:16.411951 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:32:16.411977 3798194 memory.go:184] no items to output this cycle I1122 08:32:16.412140 3798194 cpu.go:275] no items to output this cycle E1122 08:32:26.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:32:26.412892 3798194 memory.go:191] Add success. I1122 08:32:26.417158 3798194 cpu.go:282] Add success. I1122 08:32:26.441756 3798194 net.go:642] Add success. I1122 08:32:26.443185 3798194 net.go:778] primary dev: ETH0 I1122 08:32:26.443206 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:32:26.443224 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:32:29.449989 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:32:29.450073 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:32:29.450100 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:32:36.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:32:36.412256 3798194 memory.go:184] no items to output this cycle I1122 08:32:36.412399 3798194 cpu.go:275] no items to output this cycle E1122 08:32:46.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:32:46.413257 3798194 memory.go:184] no items to output this cycle I1122 08:32:46.413410 3798194 cpu.go:275] no items to output this cycle E1122 08:32:56.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:32:56.413263 3798194 memory.go:191] Add success. W1122 08:32:56.413296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:32:56.413307 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:32:56.413311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:32:56.417812 3798194 cpu.go:282] Add success. I1122 08:32:56.441227 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:32:56.441797 3798194 net.go:642] Add success. I1122 08:32:56.443761 3798194 net.go:778] primary dev: ETH0 I1122 08:32:56.443775 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:32:56.443788 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:32:57.451247 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:32:57.451413 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:32:57.451469 3798194 disk_worker.go:723] disk space is not compliant W1122 08:32:57.451473 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:32:57.464141 3798194 disk_worker.go:509] system disk:vda1 I1122 08:32:57.464190 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:32:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:32:59.450397 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:32:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:32:59.450503 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:32:59.471038 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:33:03.337137 3798194 disk_info.go:125] begin check local disk info of client I1122 08:33:03.340951 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:33:03.340960 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046cdc0 0xc00046ce00] E1122 08:33:06.411956 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:33:06.411980 3798194 memory.go:184] no items to output this cycle I1122 08:33:06.412258 3798194 cpu.go:275] no items to output this cycle E1122 08:33:16.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:33:16.412258 3798194 memory.go:184] no items to output this cycle I1122 08:33:16.412408 3798194 cpu.go:275] no items to output this cycle I1122 08:33:17.665360 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 08:33:17.665368 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 08:33:26.412232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:33:26.412267 3798194 memory.go:191] Add success. I1122 08:33:26.416722 3798194 cpu.go:282] Add success. I1122 08:33:26.442267 3798194 net.go:642] Add success. I1122 08:33:26.443796 3798194 net.go:778] primary dev: ETH0 I1122 08:33:26.443810 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:33:26.443825 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:33:29.450011 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:33:29.450082 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:33:29.450125 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:33:36.411944 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:33:36.411966 3798194 memory.go:184] no items to output this cycle I1122 08:33:36.412104 3798194 cpu.go:275] no items to output this cycle E1122 08:33:46.413223 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:33:46.413248 3798194 memory.go:184] no items to output this cycle I1122 08:33:46.413398 3798194 cpu.go:275] no items to output this cycle E1122 08:33:56.412049 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:33:56.412088 3798194 memory.go:191] Add success. W1122 08:33:56.412141 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:33:56.412156 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:33:56.412159 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:33:56.416465 3798194 cpu.go:282] Add success. I1122 08:33:56.441871 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:33:56.442384 3798194 net.go:642] Add success. I1122 08:33:56.444007 3798194 net.go:778] primary dev: ETH0 I1122 08:33:56.444021 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:33:56.444035 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:33:57.450628 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:33:57.450829 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:33:57.450840 3798194 disk_worker.go:723] disk space is not compliant W1122 08:33:57.450844 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:33:57.459232 3798194 disk_worker.go:509] system disk:vda1 I1122 08:33:57.459273 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:33:58.446008 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:33:59.450025 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:33:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:33:59.450138 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:33:59.471657 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:34:03.345433 3798194 disk_info.go:125] begin check local disk info of client I1122 08:34:03.349361 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:34:03.349369 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036e5c0 0xc00036e600] E1122 08:34:06.411959 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:34:06.411983 3798194 memory.go:184] no items to output this cycle I1122 08:34:06.412139 3798194 cpu.go:275] no items to output this cycle E1122 08:34:16.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:34:16.411985 3798194 memory.go:184] no items to output this cycle I1122 08:34:16.412146 3798194 cpu.go:275] no items to output this cycle E1122 08:34:26.412235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:34:26.412268 3798194 memory.go:191] Add success. I1122 08:34:26.416815 3798194 cpu.go:282] Add success. I1122 08:34:26.441276 3798194 net.go:642] Add success. I1122 08:34:26.443016 3798194 net.go:778] primary dev: ETH0 I1122 08:34:26.443031 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:34:26.443045 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:34:29.450185 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:34:29.450265 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:34:29.450293 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:34:36.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:34:36.411975 3798194 memory.go:184] no items to output this cycle I1122 08:34:36.412130 3798194 cpu.go:275] no items to output this cycle E1122 08:34:46.412305 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:34:46.412331 3798194 memory.go:184] no items to output this cycle I1122 08:34:46.412479 3798194 cpu.go:275] no items to output this cycle E1122 08:34:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:34:56.413267 3798194 memory.go:191] Add success. W1122 08:34:56.413298 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:34:56.413309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:34:56.413314 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:34:56.417294 3798194 cpu.go:282] Add success. I1122 08:34:56.441691 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:34:56.442184 3798194 net.go:642] Add success. I1122 08:34:56.443641 3798194 net.go:778] primary dev: ETH0 I1122 08:34:56.443657 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:34:56.443670 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:34:57.451049 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:34:57.451269 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:34:57.451279 3798194 disk_worker.go:723] disk space is not compliant W1122 08:34:57.451283 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:34:57.460074 3798194 disk_worker.go:509] system disk:vda1 I1122 08:34:57.460135 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:34:58.446381 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:34:59.450400 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:34:59.450488 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:34:59.450516 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:34:59.471012 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:35:03.349472 3798194 disk_info.go:125] begin check local disk info of client I1122 08:35:03.353305 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:35:03.353314 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00036e740 0xc00036e780] E1122 08:35:06.413244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:35:06.413264 3798194 memory.go:184] no items to output this cycle I1122 08:35:06.413411 3798194 cpu.go:275] no items to output this cycle E1122 08:35:16.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:35:16.412257 3798194 memory.go:184] no items to output this cycle I1122 08:35:16.412402 3798194 cpu.go:275] no items to output this cycle E1122 08:35:26.412233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:35:26.412272 3798194 memory.go:191] Add success. I1122 08:35:26.416717 3798194 cpu.go:282] Add success. I1122 08:35:26.441226 3798194 net.go:642] Add success. I1122 08:35:26.442608 3798194 net.go:778] primary dev: ETH0 I1122 08:35:26.442623 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:35:26.442638 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:35:29.450392 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:35:29.450468 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:35:29.450493 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:35:36.411950 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:35:36.411971 3798194 memory.go:184] no items to output this cycle I1122 08:35:36.412120 3798194 cpu.go:275] no items to output this cycle E1122 08:35:46.411987 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:35:46.412010 3798194 memory.go:184] no items to output this cycle I1122 08:35:46.412176 3798194 cpu.go:275] no items to output this cycle E1122 08:35:56.411914 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:35:56.411945 3798194 memory.go:191] Add success. W1122 08:35:56.411978 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:35:56.411990 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:35:56.411993 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:35:56.412352 3798194 cpu.go:282] Add success. I1122 08:35:56.441572 3798194 event_worker.go:152] Polling the log file for events... I1122 08:35:56.444881 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:35:56.445423 3798194 net.go:642] Add success. I1122 08:35:56.449365 3798194 net.go:778] primary dev: ETH0 I1122 08:35:56.449379 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:35:56.449391 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:35:56.449412 3798194 net.go:1064] net collector synced eni/eri metadata, refreshed_at: 1795305956, dev2_mac: { "ETH0": "fa:20:20:10:98:4f" }, dev2_type: { "ETH0": "elastic_nic" }, dev2_rdma: {}, primary_dev: ETH0 I1122 08:35:56.460499 3798194 eni_metadata.go:163] primary mac is: fa:20:20:10:98:4f, dev: ETH0 E1122 08:35:57.445460 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 08:35:57.445477 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:35:57.445667 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:35:57.445744 3798194 disk_worker.go:723] disk space is not compliant W1122 08:35:57.445747 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:35:57.448628 3798194 disk_worker.go:509] system disk:vda1 I1122 08:35:57.448665 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 08:35:58.448130 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 08:35:58.448147 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 08:35:59.448966 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 08:35:59.450039 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:35:59.450088 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:35:59.450107 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:35:59.471647 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:36:03.357143 3798194 disk_info.go:125] begin check local disk info of client I1122 08:36:03.360956 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:36:03.360965 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0004af900 0xc0004af940] E1122 08:36:06.412772 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:36:06.412795 3798194 memory.go:184] no items to output this cycle I1122 08:36:06.412946 3798194 cpu.go:275] no items to output this cycle E1122 08:36:16.411051 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:36:16.411075 3798194 memory.go:184] no items to output this cycle I1122 08:36:16.411247 3798194 cpu.go:275] no items to output this cycle I1122 08:36:17.665532 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 08:36:17.665541 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 08:36:26.412860 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:36:26.412895 3798194 memory.go:191] Add success. I1122 08:36:26.417218 3798194 cpu.go:282] Add success. I1122 08:36:26.441655 3798194 net.go:642] Add success. I1122 08:36:26.443133 3798194 net.go:778] primary dev: ETH0 I1122 08:36:26.443149 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:36:26.443161 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:36:29.450029 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:36:29.450130 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:36:29.450158 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:36:36.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:36:36.413260 3798194 memory.go:184] no items to output this cycle I1122 08:36:36.413403 3798194 cpu.go:275] no items to output this cycle E1122 08:36:46.412871 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:36:46.412894 3798194 memory.go:184] no items to output this cycle I1122 08:36:46.413041 3798194 cpu.go:275] no items to output this cycle E1122 08:36:56.412847 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:36:56.412881 3798194 memory.go:191] Add success. W1122 08:36:56.412909 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:36:56.412922 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:36:56.412926 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:36:56.417860 3798194 cpu.go:282] Add success. I1122 08:36:56.441352 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:36:56.441862 3798194 net.go:642] Add success. I1122 08:36:56.443409 3798194 net.go:778] primary dev: ETH0 I1122 08:36:56.443423 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:36:56.443435 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:36:57.450677 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:36:57.450834 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:36:57.450888 3798194 disk_worker.go:723] disk space is not compliant W1122 08:36:57.450892 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:36:57.459318 3798194 disk_worker.go:509] system disk:vda1 I1122 08:36:57.459359 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:36:58.446044 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:36:59.450133 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:36:59.450206 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:36:59.450234 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:36:59.470734 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:37:03.364138 3798194 disk_info.go:125] begin check local disk info of client I1122 08:37:03.367920 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:37:03.367928 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003fadc0 0xc0003fae00] E1122 08:37:06.412874 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:37:06.412898 3798194 memory.go:184] no items to output this cycle I1122 08:37:06.413037 3798194 cpu.go:275] no items to output this cycle E1122 08:37:16.412872 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:37:16.412900 3798194 memory.go:184] no items to output this cycle I1122 08:37:16.413059 3798194 cpu.go:275] no items to output this cycle E1122 08:37:26.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:37:26.413003 3798194 memory.go:191] Add success. I1122 08:37:26.418043 3798194 cpu.go:282] Add success. I1122 08:37:26.441727 3798194 net.go:642] Add success. I1122 08:37:26.443190 3798194 net.go:778] primary dev: ETH0 I1122 08:37:26.443206 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:37:26.443221 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:37:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:37:29.450104 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:37:29.450152 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:37:36.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:37:36.412257 3798194 memory.go:184] no items to output this cycle I1122 08:37:36.412410 3798194 cpu.go:275] no items to output this cycle E1122 08:37:46.411958 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:37:46.411981 3798194 memory.go:184] no items to output this cycle I1122 08:37:46.412138 3798194 cpu.go:275] no items to output this cycle E1122 08:37:56.411928 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:37:56.411959 3798194 memory.go:191] Add success. W1122 08:37:56.411994 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:37:56.412005 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:37:56.412009 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:37:56.412355 3798194 cpu.go:282] Add success. I1122 08:37:56.441787 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:37:56.442336 3798194 net.go:642] Add success. I1122 08:37:56.446301 3798194 net.go:778] primary dev: ETH0 I1122 08:37:56.446315 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:37:56.446328 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:37:57.450658 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:37:57.450875 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:37:57.450886 3798194 disk_worker.go:723] disk space is not compliant W1122 08:37:57.450889 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:37:57.461858 3798194 disk_worker.go:509] system disk:vda1 I1122 08:37:57.461900 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:37:58.446010 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:37:59.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:37:59.450102 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:37:59.450148 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:37:59.471720 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:38:03.372142 3798194 disk_info.go:125] begin check local disk info of client I1122 08:38:03.375973 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:38:03.375982 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00046d240 0xc00046d280] E1122 08:38:06.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:38:06.413257 3798194 memory.go:184] no items to output this cycle I1122 08:38:06.413400 3798194 cpu.go:275] no items to output this cycle E1122 08:38:16.412751 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:38:16.412773 3798194 memory.go:184] no items to output this cycle I1122 08:38:16.412924 3798194 cpu.go:275] no items to output this cycle E1122 08:38:26.413278 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:38:26.413320 3798194 memory.go:191] Add success. I1122 08:38:26.417534 3798194 cpu.go:282] Add success. I1122 08:38:26.442068 3798194 net.go:642] Add success. I1122 08:38:26.443429 3798194 net.go:778] primary dev: ETH0 I1122 08:38:26.443443 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:38:26.443461 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:38:29.450186 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:38:29.450263 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:38:29.450289 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:38:36.412230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:38:36.412252 3798194 memory.go:184] no items to output this cycle I1122 08:38:36.412396 3798194 cpu.go:275] no items to output this cycle E1122 08:38:46.412244 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:38:46.412267 3798194 memory.go:184] no items to output this cycle I1122 08:38:46.412417 3798194 cpu.go:275] no items to output this cycle E1122 08:38:56.413235 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:38:56.413268 3798194 memory.go:191] Add success. W1122 08:38:56.413296 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:38:56.413308 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:38:56.413311 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:38:56.417646 3798194 cpu.go:282] Add success. I1122 08:38:56.440948 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:38:56.441535 3798194 net.go:642] Add success. I1122 08:38:56.443128 3798194 net.go:778] primary dev: ETH0 I1122 08:38:56.443144 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:38:56.443157 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:38:57.449249 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:38:57.449436 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:38:57.449504 3798194 disk_worker.go:723] disk space is not compliant W1122 08:38:57.449507 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:38:57.451336 3798194 disk_worker.go:509] system disk:vda1 I1122 08:38:57.451373 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:38:58.445917 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:38:59.449935 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:38:59.450006 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:38:59.450033 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:38:59.471580 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:39:03.380749 3798194 disk_info.go:125] begin check local disk info of client I1122 08:39:03.384657 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:39:03.384666 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00030e100 0xc00030e140] E1122 08:39:06.412792 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:39:06.412814 3798194 memory.go:184] no items to output this cycle I1122 08:39:06.412969 3798194 cpu.go:275] no items to output this cycle E1122 08:39:16.412228 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:39:16.412251 3798194 memory.go:184] no items to output this cycle I1122 08:39:16.412411 3798194 cpu.go:275] no items to output this cycle I1122 08:39:17.666209 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 08:39:17.666218 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 08:39:26.413229 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:39:26.413262 3798194 memory.go:191] Add success. I1122 08:39:26.417434 3798194 cpu.go:282] Add success. I1122 08:39:26.441970 3798194 net.go:642] Add success. I1122 08:39:26.443470 3798194 net.go:778] primary dev: ETH0 I1122 08:39:26.443483 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:39:26.443496 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:39:29.450387 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:39:29.450459 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:39:29.450485 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:39:36.412237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:39:36.412264 3798194 memory.go:184] no items to output this cycle I1122 08:39:36.412405 3798194 cpu.go:275] no items to output this cycle E1122 08:39:46.412334 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:39:46.412356 3798194 memory.go:184] no items to output this cycle I1122 08:39:46.412505 3798194 cpu.go:275] no items to output this cycle E1122 08:39:56.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:39:56.412885 3798194 memory.go:191] Add success. W1122 08:39:56.412911 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:39:56.412922 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:39:56.412926 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:39:56.417025 3798194 cpu.go:282] Add success. I1122 08:39:56.441549 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:39:56.442093 3798194 net.go:642] Add success. I1122 08:39:56.443562 3798194 net.go:778] primary dev: ETH0 I1122 08:39:56.443577 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:39:56.443589 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:39:57.450875 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:39:57.451095 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:39:57.451106 3798194 disk_worker.go:723] disk space is not compliant W1122 08:39:57.451125 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:39:57.459760 3798194 disk_worker.go:509] system disk:vda1 I1122 08:39:57.459803 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:39:58.446382 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:39:59.450410 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:39:59.450492 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:39:59.450518 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:39:59.471081 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:40:03.388134 3798194 disk_info.go:125] begin check local disk info of client I1122 08:40:03.391963 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:40:03.391972 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00045d200 0xc00045d240] E1122 08:40:06.412234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:40:06.412255 3798194 memory.go:184] no items to output this cycle I1122 08:40:06.412407 3798194 cpu.go:275] no items to output this cycle E1122 08:40:16.411968 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:40:16.411995 3798194 memory.go:184] no items to output this cycle I1122 08:40:16.412159 3798194 cpu.go:275] no items to output this cycle E1122 08:40:26.412243 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:40:26.412279 3798194 memory.go:191] Add success. I1122 08:40:26.417059 3798194 cpu.go:282] Add success. I1122 08:40:26.441733 3798194 net.go:642] Add success. I1122 08:40:26.443135 3798194 net.go:778] primary dev: ETH0 I1122 08:40:26.443150 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:40:26.443163 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:40:29.450012 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:40:29.450088 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:40:29.450130 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:40:36.413253 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:40:36.413275 3798194 memory.go:184] no items to output this cycle I1122 08:40:36.413417 3798194 cpu.go:275] no items to output this cycle E1122 08:40:46.412241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:40:46.412262 3798194 memory.go:184] no items to output this cycle I1122 08:40:46.412408 3798194 cpu.go:275] no items to output this cycle E1122 08:40:56.412855 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:40:56.412892 3798194 memory.go:191] Add success. W1122 08:40:56.412920 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:40:56.412931 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:40:56.412934 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:40:56.417231 3798194 cpu.go:282] Add success. I1122 08:40:56.444441 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:40:56.444977 3798194 net.go:642] Add success. I1122 08:40:56.446472 3798194 net.go:778] primary dev: ETH0 I1122 08:40:56.446486 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:40:56.446501 3798194 net.go:659] Add eni metrics success, count: 7 E1122 08:40:57.450739 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 08:40:57.450753 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:40:57.450901 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:40:57.450952 3798194 disk_worker.go:723] disk space is not compliant W1122 08:40:57.450955 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:40:57.459373 3798194 disk_worker.go:509] system disk:vda1 I1122 08:40:57.459415 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 08:40:58.448043 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 08:40:58.448057 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 08:40:59.448559 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 08:40:59.449634 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:40:59.449684 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:40:59.449705 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:40:59.471220 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:41:03.396690 3798194 disk_info.go:125] begin check local disk info of client I1122 08:41:03.400582 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:41:03.400591 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00039e480 0xc00039e4c0] E1122 08:41:06.412776 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:41:06.412806 3798194 memory.go:184] no items to output this cycle I1122 08:41:06.412955 3798194 cpu.go:275] no items to output this cycle E1122 08:41:16.412854 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:41:16.412876 3798194 memory.go:184] no items to output this cycle I1122 08:41:16.413024 3798194 cpu.go:275] no items to output this cycle E1122 08:41:26.413236 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:41:26.413268 3798194 memory.go:191] Add success. I1122 08:41:26.417680 3798194 cpu.go:282] Add success. I1122 08:41:26.441282 3798194 net.go:642] Add success. I1122 08:41:26.442723 3798194 net.go:778] primary dev: ETH0 I1122 08:41:26.442738 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:41:26.442750 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:41:29.450398 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:41:29.450475 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:41:29.450501 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:41:36.411955 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:41:36.411978 3798194 memory.go:184] no items to output this cycle I1122 08:41:36.412142 3798194 cpu.go:275] no items to output this cycle E1122 08:41:46.412334 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:41:46.412357 3798194 memory.go:184] no items to output this cycle I1122 08:41:46.412707 3798194 cpu.go:275] no items to output this cycle E1122 08:41:56.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:41:56.413270 3798194 memory.go:191] Add success. W1122 08:41:56.413299 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:41:56.413310 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:41:56.413313 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:41:56.417931 3798194 cpu.go:282] Add success. I1122 08:41:56.441423 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:41:56.441906 3798194 net.go:642] Add success. I1122 08:41:56.443383 3798194 net.go:778] primary dev: ETH0 I1122 08:41:56.443397 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:41:56.443417 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:41:57.449422 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:41:57.449605 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:41:57.449660 3798194 disk_worker.go:723] disk space is not compliant W1122 08:41:57.449663 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:41:57.451516 3798194 disk_worker.go:509] system disk:vda1 I1122 08:41:57.451555 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:41:58.446380 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:41:59.450395 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:41:59.450464 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:41:59.450492 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:41:59.471035 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:42:03.400692 3798194 disk_info.go:125] begin check local disk info of client I1122 08:42:03.404573 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:42:03.404582 3798194 disk_info.go:196] parse disk info done, disk is : [0xc000307e00 0xc000307e40] E1122 08:42:06.412858 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:42:06.412882 3798194 memory.go:184] no items to output this cycle I1122 08:42:06.413028 3798194 cpu.go:275] no items to output this cycle E1122 08:42:16.411946 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:42:16.411975 3798194 memory.go:184] no items to output this cycle I1122 08:42:16.412135 3798194 cpu.go:275] no items to output this cycle I1122 08:42:17.666382 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 08:42:17.666391 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 08:42:26.413230 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:42:26.413261 3798194 memory.go:191] Add success. I1122 08:42:26.417698 3798194 cpu.go:282] Add success. I1122 08:42:26.441297 3798194 net.go:642] Add success. I1122 08:42:26.442744 3798194 net.go:778] primary dev: ETH0 I1122 08:42:26.442758 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:42:26.442769 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:42:29.449679 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:42:29.449742 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:42:29.449770 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:42:36.412876 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:42:36.412900 3798194 memory.go:184] no items to output this cycle I1122 08:42:36.413044 3798194 cpu.go:275] no items to output this cycle E1122 08:42:46.411961 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:42:46.411986 3798194 memory.go:184] no items to output this cycle I1122 08:42:46.412151 3798194 cpu.go:275] no items to output this cycle E1122 08:42:56.411924 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:42:56.411957 3798194 memory.go:191] Add success. W1122 08:42:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:42:56.411999 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:42:56.412002 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:42:56.412344 3798194 cpu.go:282] Add success. I1122 08:42:56.441827 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:42:56.442464 3798194 net.go:642] Add success. I1122 08:42:56.446895 3798194 net.go:778] primary dev: ETH0 I1122 08:42:56.446910 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:42:56.446924 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:42:57.451611 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:42:57.451814 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:42:57.451826 3798194 disk_worker.go:723] disk space is not compliant W1122 08:42:57.451829 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:42:57.461170 3798194 disk_worker.go:509] system disk:vda1 I1122 08:42:57.461221 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:42:58.445923 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:42:59.450027 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:42:59.450129 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:42:59.450155 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:42:59.471671 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:43:03.409154 3798194 disk_info.go:125] begin check local disk info of client I1122 08:43:03.413320 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:43:03.413329 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00045d500 0xc00045d540] E1122 08:43:06.412960 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:43:06.412983 3798194 memory.go:184] no items to output this cycle I1122 08:43:06.413144 3798194 cpu.go:275] no items to output this cycle E1122 08:43:16.411966 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:43:16.411991 3798194 memory.go:184] no items to output this cycle I1122 08:43:16.412157 3798194 cpu.go:275] no items to output this cycle E1122 08:43:26.413234 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:43:26.413269 3798194 memory.go:191] Add success. I1122 08:43:26.417499 3798194 cpu.go:282] Add success. I1122 08:43:26.442036 3798194 net.go:642] Add success. I1122 08:43:26.443374 3798194 net.go:778] primary dev: ETH0 I1122 08:43:26.443389 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:43:26.443401 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:43:29.450044 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:43:29.450135 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:43:29.450162 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:43:36.411972 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:43:36.411997 3798194 memory.go:184] no items to output this cycle I1122 08:43:36.412154 3798194 cpu.go:275] no items to output this cycle E1122 08:43:46.412861 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:43:46.412883 3798194 memory.go:184] no items to output this cycle I1122 08:43:46.413029 3798194 cpu.go:275] no items to output this cycle E1122 08:43:56.413232 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:43:56.413265 3798194 memory.go:191] Add success. W1122 08:43:56.413297 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:43:56.413309 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:43:56.413312 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:43:56.417557 3798194 cpu.go:282] Add success. I1122 08:43:56.441942 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:43:56.442535 3798194 net.go:642] Add success. I1122 08:43:56.444050 3798194 net.go:778] primary dev: ETH0 I1122 08:43:56.444065 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:43:56.444076 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:43:57.451703 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:43:57.451910 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:43:57.451920 3798194 disk_worker.go:723] disk space is not compliant W1122 08:43:57.451923 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:43:57.463844 3798194 disk_worker.go:509] system disk:vda1 I1122 08:43:57.463887 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:43:58.446386 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:43:59.450394 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:43:59.450478 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:43:59.450512 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:43:59.471060 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:44:03.413823 3798194 disk_info.go:125] begin check local disk info of client I1122 08:44:03.417687 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:44:03.417696 3798194 disk_info.go:196] parse disk info done, disk is : [0xc0003f6bc0 0xc0003f6c00] E1122 08:44:06.412767 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:44:06.412790 3798194 memory.go:184] no items to output this cycle I1122 08:44:06.412931 3798194 cpu.go:275] no items to output this cycle E1122 08:44:16.412240 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:44:16.412262 3798194 memory.go:184] no items to output this cycle I1122 08:44:16.412413 3798194 cpu.go:275] no items to output this cycle E1122 08:44:26.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:44:26.413276 3798194 memory.go:191] Add success. I1122 08:44:26.417793 3798194 cpu.go:282] Add success. I1122 08:44:26.441305 3798194 net.go:642] Add success. I1122 08:44:26.442958 3798194 net.go:778] primary dev: ETH0 I1122 08:44:26.442973 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:44:26.442986 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:44:29.450451 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:44:29.450531 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:44:29.450563 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:44:36.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:44:36.413253 3798194 memory.go:184] no items to output this cycle I1122 08:44:36.413409 3798194 cpu.go:275] no items to output this cycle E1122 08:44:46.412845 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:44:46.412868 3798194 memory.go:184] no items to output this cycle I1122 08:44:46.413014 3798194 cpu.go:275] no items to output this cycle E1122 08:44:56.411912 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:44:56.411945 3798194 memory.go:191] Add success. W1122 08:44:56.411976 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:44:56.411988 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:44:56.411991 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:44:56.412326 3798194 cpu.go:282] Add success. I1122 08:44:56.441791 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:44:56.442378 3798194 net.go:642] Add success. I1122 08:44:56.446610 3798194 net.go:778] primary dev: ETH0 I1122 08:44:56.446633 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:44:56.446650 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:44:57.451225 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:44:57.451397 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:44:57.451452 3798194 disk_worker.go:723] disk space is not compliant W1122 08:44:57.451455 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:44:57.462013 3798194 disk_worker.go:509] system disk:vda1 I1122 08:44:57.462089 3798194 disk_worker.go:434] add disk info successfully, len:34 I1122 08:44:58.446002 3798194 custom_config.go:106] query custom config with name: huawei_npu I1122 08:44:59.450017 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:44:59.450097 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:44:59.450134 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:44:59.471692 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:45:03.417869 3798194 disk_info.go:125] begin check local disk info of client I1122 08:45:03.421676 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:45:03.421685 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00027f440 0xc00027f480] E1122 08:45:06.412788 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:45:06.412818 3798194 memory.go:184] no items to output this cycle I1122 08:45:06.412961 3798194 cpu.go:275] no items to output this cycle E1122 08:45:16.413233 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:45:16.413256 3798194 memory.go:184] no items to output this cycle I1122 08:45:16.413404 3798194 cpu.go:275] no items to output this cycle I1122 08:45:17.666552 3798194 nvml_worker.go:361] ERROR_LIBRARY_NOT_FOUND I1122 08:45:17.666561 3798194 nvml_worker.go:376] NVIDIA driver is not ready, try again in 1 minute. E1122 08:45:26.413312 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:45:26.413357 3798194 memory.go:191] Add success. I1122 08:45:26.417441 3798194 cpu.go:282] Add success. I1122 08:45:26.441949 3798194 net.go:642] Add success. I1122 08:45:26.443510 3798194 net.go:778] primary dev: ETH0 I1122 08:45:26.443526 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:45:26.443539 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:45:29.450021 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:45:29.450100 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:45:29.450140 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:45:36.411962 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:45:36.411986 3798194 memory.go:184] no items to output this cycle I1122 08:45:36.412141 3798194 cpu.go:275] no items to output this cycle E1122 08:45:46.412852 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:45:46.412970 3798194 memory.go:184] no items to output this cycle I1122 08:45:46.413340 3798194 cpu.go:275] no items to output this cycle E1122 08:45:56.413231 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:45:56.413265 3798194 memory.go:191] Add success. W1122 08:45:56.413291 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:45:56.413303 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:45:56.413306 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:45:56.417540 3798194 cpu.go:282] Add success. I1122 08:45:56.442187 3798194 event_worker.go:152] Polling the log file for events... I1122 08:45:56.442421 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:45:56.442992 3798194 net.go:642] Add success. I1122 08:45:56.444562 3798194 net.go:778] primary dev: ETH0 I1122 08:45:56.444577 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:45:56.444591 3798194 net.go:659] Add eni metrics success, count: 7 E1122 08:45:57.446449 3798194 gpu_controllor.go:92] DCGM discovery failed: exec: "dcgmi": executable file not found in $PATH I1122 08:45:57.446464 3798194 custom_config.go:106] query custom config with name: gpu W1122 08:45:57.446715 3798194 disk_worker.go:1029] failed to get sector size for device nsfs: lstat /sys/class/block/nsfs: no such file or directory W1122 08:45:57.446726 3798194 disk_worker.go:723] disk space is not compliant W1122 08:45:57.446729 3798194 disk_worker.go:743] disk inode is not compliant I1122 08:45:57.449525 3798194 disk_worker.go:509] system disk:vda1 I1122 08:45:57.449565 3798194 disk_worker.go:434] add disk info successfully, len:34 E1122 08:45:58.448504 3798194 npu_controller.go:127] Fail to call npu-smi, exit status 127, bash: /usr/local/sbin/npu-smi: No such file or directory I1122 08:45:58.448518 3798194 custom_config.go:106] query custom config with name: huawei_npu E1122 08:45:59.449198 3798194 rdma_controller.go:118] fail to call which hccn_too, err:exit status 1, output:which: no hccn_tool in (/usr/sbin:/usr/bin:/sbin:/bin) I1122 08:45:59.450277 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:45:59.450330 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:45:59.450351 3798194 gpu_kunlun.go:227] Add success, len:1 I1122 08:45:59.470876 3798194 disk_local_worker.go:436] Get disk info: [] I1122 08:46:03.422270 3798194 disk_info.go:125] begin check local disk info of client I1122 08:46:03.426164 3798194 disk_info.go:161] parse disk info: sr0 QM00051 vda v-9blS3uH4 I1122 08:46:03.426172 3798194 disk_info.go:196] parse disk info done, disk is : [0xc00025ef40 0xc00025ef80] E1122 08:46:06.412859 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:46:06.412881 3798194 memory.go:184] no items to output this cycle I1122 08:46:06.413038 3798194 cpu.go:275] no items to output this cycle E1122 08:46:16.413241 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:46:16.413268 3798194 memory.go:184] no items to output this cycle I1122 08:46:16.413427 3798194 cpu.go:275] no items to output this cycle E1122 08:46:26.411922 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:46:26.411957 3798194 memory.go:191] Add success. I1122 08:46:26.412309 3798194 cpu.go:282] Add success. I1122 08:46:26.441923 3798194 net.go:642] Add success. I1122 08:46:26.446159 3798194 net.go:778] primary dev: ETH0 I1122 08:46:26.446176 3798194 net.go:810] Send network stats successfully!,count is 6 I1122 08:46:26.446189 3798194 net.go:659] Add eni metrics success, count: 7 I1122 08:46:29.450018 3798194 gpu_kunlun.go:233] start to get kun lun stat by xpuml I1122 08:46:29.450096 3798194 gpu_kunlun.go:354] start to get kun lun stat by xpu_smi I1122 08:46:29.450136 3798194 gpu_kunlun.go:227] Add success, len:1 E1122 08:46:36.413238 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:46:36.413261 3798194 memory.go:184] no items to output this cycle I1122 08:46:36.413401 3798194 cpu.go:275] no items to output this cycle E1122 08:46:46.413237 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:46:46.413259 3798194 memory.go:184] no items to output this cycle I1122 08:46:46.413408 3798194 cpu.go:275] no items to output this cycle E1122 08:46:56.412850 3798194 memory.go:175] total swap size <= 0, can not cal swap used percent I1122 08:46:56.412883 3798194 memory.go:191] Add success. W1122 08:46:56.412915 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ce_count W1122 08:46:56.412926 3798194 memory.go:428] No files found matching pattern: /sys/devices/system/edac/mc/mc*/ue_count I1122 08:46:56.412929 3798194 memory.go:414] ECC metrics: ce_delta=0, ue_delta=0 I1122 08:46:56.417129 3798194 cpu.go:282] Add success. I1122 08:46:56.441531 3798194 controller.go:85] Send host metrics successfully, count is 5 I1122 08:46:56.442054 3798194 net.go:642] Add success.